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Part  I 


What  is  in  This  Report? 

This  document,  combined  with  associated  code  (sent  electronically  earlier  under  separate  cover,  and  —  in 
the  case  of  Athena  —  acquired  online),  constitutes  the  final  report  for: 

Project  name:  “New  Architectures,  Algorithms,  and  Designs  that  Lead  to  Implemented  Ma¬ 
chine  Reasoning  over  Knowledge  in  Epistemic  &  Deontic  Formats,  in  the  Service  of  Advanced 
Wargaming” 

Pis:  Selmer  Bringsjord,  Konstantine  Arkoudas,  Yingrui  Yang.  (POC:  Bringsjord.) 

Grant  total:  $75,000  (includes  overhead) 

The  structure  of  this  report  is  as  follow: 

•  Summary  of  Work  Performed,  Indexed  to  SOW. 

•  Paper  =1,  on  multi-agent  reasoning  in  connection  with  epistemic  logic:  “Metareasoning  for  Multi- 
Agent  Epistemic  Logics."]]] 

Hereafter  referred  to  as  simply  ‘LNAI  paper.’ 

•  Paper  #2,  on  an  advanced  synthetic  character  (E)  in  connection  with  the  RASCALS  cognitive  archi¬ 
tecture:  “Advanced  Synthetic  Characters,  Evil,  and  E.”]^ 

Hereafter  referred  to  as  simply  ‘GameOn-2005  paper.’ 

•  Paper  #3,  on  mechanizing  deontic  logic:  “Toward  Ethical  Robots  via  Mechanized  Deontic  Logic” 

Hereafter  referred  to  as  simply  ‘AAAI-FS  paper.’ 

•  Paper  #4,  on  generalizing  our  mechanizing  of  deontic  logic:  ,  about  to  appear  in  IEEE  Intelligent 
Systems [] 

Hereafter  referred  to  as  simply  ‘IEEE  paper.’ 

•  Athena  tutorial  (in-person  demos  of  Athena  at  AFRL  to  follow  in  May  2006,  tentatively  scheduled 
specifically  for  the  week  of  May  8) 

•  Explanation  of  associated  code  (in-person  demos  at  AFRL  took  place  in  May  2006).  Materials  (Win¬ 
dows  executable  of  1-agent  derived  from  Slate,  tutorial  transcript  of  simple  session  recorded  demo  of 
the  executable  as  a  movie,  and  two  ath  files)  are  available  at: 


http://www.cogsci.rpi.edu/research/rair/wargaming/ 


1  The  full  reference  is:  Arkoudas,  A.  Sz  Bringsjord.  S.  (2005)  “Metareasoning  for  Multi-agent  Epistemic  Logics,”  in  Lecture 
Notes  in  Artificial  Intelligence  (LNAI),  (New  York,  NY:  Springer-Verlag),  pp.  111-125. 

2The  full  reference  is:  Bringsjord,  S.,  Khemlani,  S.,  Arkoudas,  K.,  McEvoy,  C.,  Destefano,  M.,  Daigle,  M.  (2005)  “Advanced 
Synthetic  Characters,  Evil,  and  E,”  Game- On  2005,  6th  International  Conference  on  Intelligent  Games  and  Simulation, 
(Ghent-Zwijnaarde,  Belgium:  European  Simulation  Society),  pp.  31-39. 

3 The  paper  is  in  production  at  present. 
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Part  II 


Work  Performed,  Indexed  to  SOW 

1  Overall  Objective 

The  overall  objective  of  this  project,  pulled  verbatim  from  the  SOW: 

1.1  The  objective  of  this  effort  is  to  investigate  new  architectures,  algorithms,  and  designs  to  lead 
to:  implemented  machine  reasoning  over  knowledge  in  expressive  formats  that  include  doxastic 
(=  epistemic)  and  deontic  operators;  understanding  of  the  tractability  of  using  such  implemen¬ 
tations  in  a  multi-agent  setting;  and  transfer  of  such  architectures,  algorithms,  and  eventual 
implementations  into  other  relevant  DoD-related  efforts. 

Numerous  such  investigations  have  been  conducted.  The  RASCALS  cognitive  architecture  has  been  devel¬ 
oped  further,  in  connection  with  the  synthetic  character  known  simply  as  ‘E’;  new  algorithms  have  been 
devised  in  the  area  of  mechanized  epistemic  and  deontic  logics;  and  the  basic  design  for  a  logicist  intelli¬ 
gent  agent  (or  just  an  1-agent),  callable  from  other  systems,  has  been  implemented.  Corresponding  to  these 
achievements,  which  are  associated  with  the  more  specific  points  in  the  SOW  (as  explained  below),  are  pub¬ 
lished  papers  and  code  made  available  to  AFRL.  The  papers  are  included  in  this  document,  and  the  code 
for  1-agents,  contextualized  in  this  document,  is  provided  at 


http://www.cogsci.rpi.edu/research/rair/wargaming/ 


At  this  location,  a  Windows  executable  is  provided,  as  well  as  a  transcript  of  a  simple  session,  a  recorded 
tutorial  demonstration,  and  two  ath  files. 


2  Technical  Requirements  from  the  SOW 

2.1  Requirement  4.1 

4.1  The  contractor  shall  design,  develop,  document,  demonstrate,  and  deliver  a  machine  reasoning 
capability  to  help  support  construction  of  cognitively  robust  intelligent  agents;  specifically,  this 
capability  will  allow  agents  to  reason  over  doxastic  and  deontic  information.  This  capability  will 
be  realized  in  the  contractor’s  pre-existing  tools  and  systems  (“contractors  systems”).  These 
pre-existing  systems  are  owned  by  the  contractor,  and  include:  Athena,  MARMML,  Slate,  and 
RASCALS.  Other  pre-existing  systems  will  be  used  as  well,  and  are  in  the  public  domain  (e.g., 
SNARK). 

The  capability  in  question  is  detailed  in  two  papers  included  here  (LNAI  and  AAAI-FS/IEEE).  Reasoning 
over  doxastic  information  is  detailed  in  the  LNAI  paper.  Reasoning  over  deontic  information  is  detailed 
in  the  AAAI-FS/IEEE  paper.  Corresponding  ath  files  are  provided.  This  brings  us  to  the  more  specific 
sub-requirement  here: 

2.1.1  Requirement  4.1.1 

Rendering  Scenarios  Expressed  Doxastic  Systems  (e.g.,  KD4-5)  in  Computational  Form  via  Logic- 
Based  AI  Techniques.  The  contractor  shall  develop  the  theoretical  constructions  (architectures, 
algorithms,  designs,  etc.)  necessary  for  the  computational  implementation,  in  the  contractor’s 
systems,  of  test  scenarios  expressed  in  the  modal  logic  KD45  (and/or  other  such  logics)  of  belief 
and  knowledge.  This  logic  allows  for  reasoning  over  doxastic  information,  which  is  information 
about  what  agents  believe  and  know.  The  contractor  shall  provide  a  method  to  address  the 
technical  problem  known  as  “logical  omniscience.” 
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The  theoretical  constructions  called  for  in  4.1.1  have  been  invented,  and  are  presented  in  the  LNAI  paper. 
The  implementation  has  been  accomplished  as  well;  it  is  reported  in  the  LNAI  paper,  and,  again,  the 
associated  code  is  provided  on  the  RAIR  Lab’s  web  site  (see  Part  VIII I .  The  next  sub-requirement  under 
4.1  is: 


2.1.2  Requirement  4.1.2 

Rendering  Scenarios  Expressed  Deontic  Systems  (e.g.,  DSDL3)  in  Computational  Form  via  Logic- 
Based  AI  Techniques.  The  contractor  shall  develop  the  theoretical  constructions  necessary  for 
the  computational  implementation,  in  the  contractors  systems,  of  test  scenarios  expressed  in  the 
modal  logic  DSDL3  (Lewis  1974)  (and/or  other  such  logics)  of  obligation.  This  logic  allows  for 
reasoning  over  deontic  information,  which  is  information  about  the  moral  status  of  actions.  The 
contractor  shall  address  the  technical  problem  known  as  “adequacy”  (or  conditional  obligation). 


The  theoretical  constructions  alluded  to  here  have  been  invented,  and  are  specified  in  the  AAAI/IEEE  paper. 
In  addition,  the  implementation  has  been  accomplished.  The  relevant  Athena  code  is  provided  on  the  RAIR 
Lab  web  site  (see  Part  VIII  l . 


2.1.3  Requirement  4.1.3 

Long-Term  Use  by  AFRL.  The  contractor  will  investigate  the  design  of  an  application  program¬ 
ming  interface  (API)  that  allows  developers  working  in  the  Java  programming  language  at  AFRL 
to  exploit  the  capability  referred  to  in  section  4.1.  Such  an  API  will  make  possible  the  long-term 
use  of  this  capability  in  conjunction  with  the  contractors  systems  at  AFRL.  Members  of  AFRL’s 
Third  Generation  Wargaming  Group  (3GWG)  will  advise  the  contractor  about  the  conditions 
the  API  must  satisfy,  and  will  make  available  its  Java-based  systems,  and  specifications  thereof, 
to  the  contractor,  in  order  to  enable  the  design  of  API  by  the  contractor. 

Conditions  that  the  API  would  need  to  satisfy  have  not  been  provided,  and  the  Java-based  systems,  and 
specifications  thereof,  have  not  been  provided  by  AFRL’s  3GWG.  (Are  the  systems  in  question  still  under 
development  at  AFRL  Rome?)  However,  the  RAIR  Lab  has  nonetheless  designed  and  built  a  number  of 
functions  that  would  be  central  parts  of  any  calls  out  from  3GWG  Java-based  systems  to  our  reasoning 
technology.  These  functions  are  provided  on  the  RAIR  Lab’s  web  site,  and  can  be  demonstrated  and 
explained  in  person  in  the  planned  upcoming  visit  on  or  shortly  after  May  8.  Calls  out  from  any  Java-based 
system  should  follow  the  interoperability  standards  currently  in  place  in  DTO,  which  are  based  on  the  ISO 
Common  Logic  standard,  and  are  gaining  steam  as  an  across-the-board  standard  for  DoD  R&D.  Readers  are 
once  again  referred  to  Part  |VIII| 


2.1.4  Requirement  4.1.4 

Testbed  Development.  The  contractor  shall  document  the  appropriate  set  of  test  scenarios  referred 
to  in  sections  2.1.1  and  2.1.3,  coordinated  with  the  Government,  for  the  reasoning  technology 
developed.  Some  of  these  test  scenarios  will  involve  doxastic  information;  some  will  involve 
deontic  information. 

•  4. 1.4.1  The  contractor  shall  deliver  all  applications  developed  and  associated  software  im¬ 
plementations  of  the  modal  logics  described  in  the  form  of  computer  code  executable  in  the 
contractor’s  systems. 

•  4. 1.4. 2  The  contractor  shall  develop  and  deliver  a  detailed  technical  tutorial  and  maintenance 
document  for  the  contractors  systems. 

The  test  scenarios  are  detailed  in  the  LNAI  (doxastic)  and  AAAI/IEEE  (deontic)  papers.  Once  again,  code 
is  available  on  the  RAIR  Lab’s  web  site.  Regarding  tutorial  information,  a  full  Athena  tutorial  is  included 
in  this  document.  In-person  tutorials  will  be  provided  as  well  (tentatively  planned  for  the  week  of  May  8 
2006). 
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Metareasoning  for  multi-agent  epistemic  logics 


Konst antine  Arkoudas  and  Selmer  Bringsjord 
RPI 

{arkouk,brings}@rpi . edu 


Abstract.  We  present  an  encoding  of  a  sequent  calculus  for  a  multi- 
agent  epistemic  logic  in  Athena,  an  interactive  theorem  proving  system 
for  many-sorted  first-order  logic.  We  then  use  Athena  as  a  metalanguage 
in  order  to  reason  about  the  multi-agent  logic  an  as  object  language. 
This  facilitates  theorem  proving  in  the  multi-agent  logic  in  several  ways. 
First,  it  lets  us  marshal  the  highly  efficient  theorem  provers  for  clas¬ 
sical  first-order  logic  that  are  integrated  with  Athena  for  the  purpose 
of  doing  proofs  in  the  multi-agent  logic.  Second,  unlike  model-theoretic 
embeddings  of  modal  logics  into  classical  first-order  logic,  our  proofs  are 
directly  convertible  into  native  epistemic  logic  proofs.  Third,  because  we 
are  able  to  quantify  over  propositions  and  agents,  we  get  much  of  the 
generality  and  power  of  higher-order  logic  even  though  we  are  in  a  first- 
order  setting.  Finally,  we  are  able  to  use  Athena’s  versatile  tactics  for 
proof  automation  in  the  multi-agent  logic.  We  illustrate  by  developing  a 
tactic  for  solving  the  generalized  version  of  the  wise  men  problem. 


1  Introduction 

Multi-agent  modal  logics  are  widely  used  in  Computer  Science  and  AI.  Multi¬ 
agent  epistemic  logics,  in  particular,  have  found  applications  in  fields  ranging 
from  AI  domains  such  as  robotics,  planning,  and  motivation  analysis  in  natu¬ 
ral  language  [13];  to  negotiation  and  game  theory  in  economics;  to  distributed 
systems  analysis  and  protocol  authentication  in  computer  security  [16,31].  The 
reason  is  simple — intelligent  agents  must  be  able  to  reason  about  knowledge.  It  is 
therefore  important  to  have  efficient  means  for  performing  machine  reasoning  in 
such  logics.  While  the  validity  problems  for  most  propositional  modal  logics  are 
of  intractable  theoretical  complexity1 ,  several  approaches  have  been  investigated 
in  recent  years  that  have  resulted  in  systems  that  appear  to  work  well  in  prac¬ 
tice.  These  approaches  include  tableau-based  provers,  SAT-based  algorithms, 
and  translations  to  first-order  logic  coupled  with  the  use  of  resolution-based  the¬ 
orem  provers.  Some  representative  systems  are  Fact  [24],  KSATC  [14],  TA  [25], 
LWB  [23],  and  MSPASS  [38]. 

Tranlation-based  approaches  (such  as  that  of  MSPASS)  have  the  advantage 
of  leveraging  the  tremendous  implementation  progress  that  has  occurred  over 

1  For  instance,  the  validity  problem  for  multi-agent  propositional  epistemic  logic  is 
PSPACE-complete  [18];  adding  a  common  knowledge  operator  makes  the  problem 
EXPTIME-complete  [21]. 
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the  last  few  decades  in  first-order  theorem  proving.  Soundness  and  completeness 
are  ensured  by  the  soundness  and  completeness  of  the  resolution  prover  (once  the 
soundness  and  completeness  of  the  translation  have  been  shown) ,  while  a  decision 
procedure  is  automatically  obtained  for  any  modal  logic  that  can  be  translated 
into  a  decidable  fragment  of  first-order  logic  (such  as  the  two- variable  fragment). 
Furthermore,  Kripke  frames  are  first-order  definable  [17],  so  translating  from  a 
modal  setting  to  the  classical  first-order  setting  is  fairly  straightforward.  For 
instance,  the  well-known  formula  [DP  A  d(P  =>  Q)\  =>  OQ  becomes 

V  Wi  .  [(V  w2  .  R(w i,w2)  ^-P{w 2))  A 

(V  w2  •  R(w i,  w2)  =>  P(w2 )  ^Q(w 2))]  =>  (V  w2  •  R(wi,w2)  =>  Q(w2 )) 

Here  the  variables  w  1  and  w2  range  over  possible  worlds,  and  the  relation  R 
represents  Kripke’s  accessibility  relation.  A  constant  propositional  atom  P  in 
the  modal  language  becomes  a  unary  predicate  P(w)  that  holds  (or  not)  for  a 
given  world  w. 

This  is  the  classical  translation  of  modal  logic  into  first-order  logic  [18],  and 
we  might  say  that  it  is  a  semantic  embedding,  since  the  Kripke  semantics  of 
the  modal  language  are  explicitly  encoded  in  the  translated  result.  This  is,  for 
instance,  the  approach  taken  by  McCarthy  in  his  “Formalizing  two  puzzles  in¬ 
volving  knowledge”  [30].  A  drawback  of  this  approach  is  that  proofs  produced 
in  the  translated  setting  are  difficult  to  convert  back  into  a  form  that  makes 
sense  for  the  user  in  the  original  modal  setting,  altough  alternative  translation 
techniques  such  as  the  functional  translation  to  path  logic  can  alleviate  this  issue 
in  some  cases  [39].  Another  drawback  is  that  if  a  result  is  not  obtained  within 
a  reasonable  amount  of  time  (which  is  almost  certain  to  happen  when  no  deci¬ 
sion  procedure  is  available,  as  in  first-order  modal  logics),  then  a  batch-oriented 
prover  is  of  little  help  to  the  user  due  to  its  “low  bandwidth  of  interaction”  [12]. 
Much  greater  flexibilitlnteractive  proof  systems  such  as  PVS  [37],  HOL  [20],  Is¬ 
abelle  [34],  and  Athena  [2]  that  offer  tactics,  facilities  for  goal  decomposition  and 
computation, 

In  this  paper  we  explore  another  approach:  We  embed  a  multi-agent  epis¬ 
temic  logic  into  many-sorted  first-order  logic  in  a  proof-theoretic  rather  than 
in  a  model-theoretic  way.  2  Specifically,  we  use  the  interactive  theorem  proving 
system  Athena  (which  is  briefly  reviewed  in  the  Appendix)  to  encode  the  for¬ 
mulas  of  the  epistemic  logic  along  with  the  inference  rules  of  a  sequent  calculus 
for  it.  Hence  first-order  logic  becomes  our  metalanguage  and  the  epistemic  logic 
becomes  our  object  language.  We  then  use  standard  first-order  reasoning  (our 
metalanguage)  to  reason  about  proofs  in  the  object  logic.  In  effect,  we  end  up 
reasoning  about  reasoning — hence  the  term  metareasoning.  Since  our  metarea¬ 
soning  occurs  at  the  standard  first-order  level,  we  are  free  to  leverage  existing 
theorem-proving  systems  for  automated  deduction.  In  particular,  we  make  heavy 

2  This  paper  treats  a  propositional  logic  of  knowledge,  but  the  technique  can  be  readily 
applied  to  full  first-order  multi-agent  epistemic  logic,  and  indeed  to  hybrid  multi¬ 
modal  logics,  e.g.,  combination  logics  for  temporal  and  epistemic  reasoning. 
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use  of  Vampire  [41],  a  cutting-edge  resolution-based  prover  that  is  seamlessly  in¬ 
tegrated  with  Athena. 

Our  approach  has  two  additional  advantages.  First,  it  is  trivial  to  translate 
the  constructed  proofs  into  modal  form,  since  the  Athena  proofs  are  already 
about  proofs  in  the  modal  logic.  Second,  because  the  abstract  syntax  of  the  epis- 
temic  logic  is  explicitly  encoded  in  Athena,  we  can  quantify  over  propositions, 
sequents,  and  agents.  Accordingly,  we  get  the  generalization  benefits  of  higher- 
order  logic  even  in  a  first-order  setting.  This  can  result  in  significant  efficiency 
improvements.  For  instance,  in  solving  the  generalized  wise  men  puzzle  it  is  nec¬ 
essary  at  some  point  to  derive  the  conclusion  M2  V  •  •  •  V  Mn  from  the  three 
premises  Ka(^(M2  V  •  •  •  V  Mn)  =k  Mf),  and 


-(M2  V  •  •  •  V  M„)  =>  Ka(-i(M2  V  •  •  •  V  Mn)) 


where  Mi, . . . ,  Mn  are  atomic  propositions  and  a  is  an  epistemic  agent,  n  >  1. 
In  the  absence  of  an  explicit  embedding  of  the  epistemic  logic,  this  would  have 
to  be  done  with  a  tactic  that  accepted  a  list  of  propositions  [Mi  •  •  •  Mn]  as  input 
and  performed  the  appropriate  deduction  dynamically,  which  would  require  an 
amount  of  effort  quadratic  in  the  length  of  the  list.  By  contrast,  in  our  approach 
we  are  able  to  formulate  and  prove  a  “higher-order”  lemma  stating 

V  P,Q, a  .  {-'Ka(P),KahQ^P)^Q^KahQ)}^Q 

Obtaining  the  desired  conclusion  for  any  given  Mi, . . . ,  Mn  then  becomes  a  mat¬ 
ter  of  instantiating  this  lemma  with  P  i— >  Mi  and  Q  i— >  M2  V  •  •  ■  V  Mn.  We  have 
thus  reduced  the  asymptotic  complexity  of  our  task  from  quadratic  time  to  con¬ 
stant  time. 

But  perhaps  the  most  distinguishing  aspect  of  our  work  is  our  emphasis  on 
tactics.  Tactics  are  proof  algorithms,  which,  unlike  conventional  algorithms,  are 
guaranteed  to  produce  sound  results.  That  is,  if  and  when  a  tactic  outputs  a 
result  P  that  it  claims  to  be  a  theorem,  we  can  be  assured  that  P  is  indeed  a 
theorem.  Tactics  are  widely  used  for  proof  automation  in  first-  and  higher-order 
proof  systems  such  as  HOL  [20]  and  Isabelle  [34].  In  Athena  tactics  are  called 
methods,  and  are  particularly  easy  to  formulate  owing  to  Athena’s  Fitch-style 
natural  deduction  system  and  its  assumption-base  semantics  [3[.  A  major  goal  of 
our  research  is  to  find  out  how  easy — or  difficult — it  may  be  to  automate  multi¬ 
agent  modal  logic  proofs  with  tactics.  Our  aim  is  not  to  obtain  a  completely 
automatic  decision  procedure  for  a  certain  logic  (or  class  of  logics) ,  but  rather  to 
enable  efficient  interactive — i.e.,  semi-automatic — theorem  proving  in  such  logics 
for  challenging  problems  that  are  beyond  the  scope  of  completely  automatic 
provers.  In  this  paper  we  formulate  an  Athena  tactic  for  solving  the  generalized 
version  of  the  wise  men  problem  (for  any  given  number  of  wise  men) .  The  relative 
ease  with  which  this  method  was  formulated  is  encouraging. 

The  remainder  of  this  paper  is  structured  as  follows.  In  the  next  section  we 
present  a  sequent  calculus  for  the  epistemic  logic  that  we  will  be  encoding.  In 
Section  3  we  present  the  wise  men  puzzle  and  formulate  an  algorithm  for  solving 
the  generalized  version  of  it  in  the  sequent  calculus  of  Section  2.  In  Section  4 
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PhP _ PhQ  [A-/]  rhPAQ  [A-Si]  PhPA<?  [A-E2] 

pfpaq  r\-p  r\-Q 


r\-p 

PhPvQ 


[V-/i] 


PHQ 

n-PvQ 


[v-/2] 


r  \-  Py  V  p2  r.  Pi  hQ  r,P2PQ  [v_£] 
r\-Q 


r-.P'rQ  P'rP^Q  pi-p  [=>_E] 

PhP=>Q  PhQ 


Ph^~lP  h -E\  P’Ph±  [-.-/]  _  [Reflex] 

PhP  PI--.P  P,  P  h  P 


PhP  [ Dilution ]  P  h  P  A  ^P  [J _-/]  _  [T-7] 

PUP'hP  Phi  PhT 


Fig.  1.  Inference  rules  for  the  propositional  connectives. 


we  discuss  the  Athena  encoding  of  the  epistemic  logic  and  present  the  Athena 
method  for  solving  the  generalized  wise  men  problem.  Finally,  in  Section  5  we 
consider  related  work. 


2  A  sequent  formulation  of  a  multi-agent  epistemic  logic 

We  will  use  the  letters  P,  Q,  R, . . to  designate  arbitrary  propositions,  built 
according  to  the  following  abstract  grammar: 

P::=A|T|_L|^P|PAQ|PV(2|P=><2|  Ka(P)  \  C(P) 

where  A  and  a  range  over  a  countable  set  of  atomic  propositions  (“atoms”)  and 
a  primitive  domain  of  agents,  respectively.  Propositions  of  the  form  Ka(P)  and 
C(P)  are  read  as  follows: 

Ka{P) ■  agent  a  knows  proposition  P 

C(P):  it  is  common  knowledge  that  P  holds 

By  a  context  we  will  mean  a  finite  set  of  propositions.  We  will  use  the  letter 
P  to  denote  contexts.  We  define  a  sequent  as  an  ordered  pair  (P,  P)  consisting  of 
a  context  P  and  a  proposition  P.  A  more  suggestive  notation  for  such  a  sequent 
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-  [K]  -  [T] 

P  b  [Ka(P  =>  Q)}  =>  [Ka(P)  =>  Ka(Q)]  r  b  A'a(P)  =>  P 


0hP  [C-/1  _  [C-E] 

r  b  C{P)  r  h  C(P)  A'a(P) 


-  [CK]  -  [R\ 

r  h  [c(p  Q)]  =>  [C(P)  =*  c{Q)\  r  h  c(P)  =>  c(a'q(p)) 


Fig.  2.  Inference  rules  for  the  epistemic  operators. 


is  r  b  P.  Intuitively,  this  is  a  judgment  stating  that  P  follows  from  r.  We  will 
write  P,r  (or  P,  P)  as  an  abbreviation  for  PU  {P}.  The  sequent  calculus  that 
we  will  use  consists  of  a  collection  of  inference  rules  for  deriving  judgments  of 
the  form  PhP.  Figure  1  shows  the  inference  rules  that  deal  with  the  standard 
propositional  connectives.  This  part  is  standard  (e.g.,  it  is  very  similar  to  the 
sequent  calculus  of  Ebbinghaus  et  al.  [15]).  In  addition,  we  have  some  rules 
pertaining  to  Ka  and  C,  shown  in  Figure  2. 

Rule  [K]  is  the  sequent  formulation  of  the  well-known  Kripke  axiom  stating 
that  the  knowledge  operator  distributes  over  conditionals.  Rule  [Ck]  is  the  cor¬ 
responding  principle  for  the  common  knowledge  operator.  Rule  [T\  is  the  “truth 
axiom” :  an  agent  cannot  know  false  propositions.  Rule  [ Ci ]  is  an  introduction 
rule  for  common  knowledge:  if  a  proposition  P  follows  from  the  empty  set  of 
hypotheses,  i.e. ,  if  it  is  a  tautology,  then  it  is  commonly  known.  This  is  the 
common-knowledge  version  of  the  “omniscience  axiom”  for  single-agent  knowl¬ 
edge  which  says  that  P  b  Ka(P)  can  be  derived  from  0  b  P.  We  do  not  need  to 
postulate  that  axiom  in  our  formulation,  since  it  follows  from  [C-I]  and  [C-E\. 
The  latter  says  that  if  it  is  common  knowledge  that  P  then  any  (every)  agent 
knows  P,  while  [P]  says  that  if  it  is  common  knowledge  that  P  then  it  is  common 
knowledge  that  (any)  agent  a  knows  it.  [P]  is  a  reiteration  rule  that  allows  us  to 
capture  the  recursive  behavior  of  C,  which  is  usually  expressed  via  the  so-called 
“induction  axiom” 

C(P=>E(P))^[P=>C(P)] 

where  E  is  the  shared  knowledge  operator.  Since  we  do  not  need  E  for  our 
purposes,  we  omit  its  formalization  and  “unfold”  C  via  rule  [P]  instead. 

We  state  a  few  lemmas  that  will  come  handy  later: 

Lemma  1  (Cut).  If  I\  b  P\  and  P2,  Pi  b  P2  then  Pi  U  P2  b  P2. 

Proof:  Assume  Pi  b  Pi  and  P2,  Pi  b  P2.  Then,  by  [=>-/],  we  get  P2  b  Pi  =>  P2. 
Further,  by  dilution,  we  have  Pi  U  P2  b  Pi  =>  P2  and  Pi  U  P2  b  Px.  Hence,  by 
[=>-P],  we  obtain  Pi  U  P2  b  P2.  □ 

The  proofs  of  the  remaining  lemmas  are  equally  simple  exercises: 
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Lemma  2  (^-transitivity).  If  P  f-P\  ->  P>  and  F  b  P2  =>  P3  then  F  b  Pi  =>  P3. 
Lemma  3  (contrapositive).  If  F  b  P  =>-  Q  then  F  I — <Q  =>-  -1 P. 

Lemma  4.  (a)  0  b  (Pi  V  P2)  =4-  (-1P2  =b  Pi);  and  (b)  P  b  C(P2)  whenever 
0hPi=^P2  andPbC(Pi). 

Lemma  5.  For  all  P,  Q,  and  F,  F  b  [C(P)  A  C(Q)\  =>  C(P  A  Q). 

3  The  generalized  wise  men  puzzle 

Consider  first  the  three-men  version  of  the  puzzle: 

Three  wise  men  are  told  by  their  king  that  at  least  one  of  them  has  a 
white  spot  on  his  forehead.  In  reality,  all  three  have  white  spots  on  their 
foreheads.  We  assume  that  each  wise  man  can  see  the  others’  foreheads 
but  not  his  own,  and  thus  each  knows  whether  the  others  have  white 
spots.  Suppose  we  are  told  that  the  first  wise  man  says,  “I  do  not  know 
whether  I  have  a  white  spot,”  and  that  the  second  wise  man  then  says, 

“I  also  do  not  know  whether  I  have  a  white  spot.”  Now  consider  the 
following  question:  Does  the  third  wise  man  now  know  whether  or  not 
he  has  a  white  spot?  If  so,  what  does  he  know,  that  he  has  one  or  doesn’t 
have  one? 

This  version  is  essentially  identical  to  the  muddy- children  puzzle,  the  only 
difference  being  that  the  declarations  of  the  wise  men  are  made  sequentially, 
whereas  in  the  muddy-children  puzzle,  the  children  proclaim  what  they  know 
(or  not  know)  in  parallel  at  every  round. 

In  the  generalized  version  of  the  puzzle  we  have  an  arbitrary  number  n  +  1 
of  wise  men  w\, . . . ,  w„+i,  n  >  1.  They  are  told  by  their  king  that  at  least  one 
them  has  a  white  spot  on  his  forehead.  Again,  in  actuality  they  all  do.  And  they 
can  all  see  one  another’s  foreheads,  but  not  their  own.  Supposing  that  each  of 
the  first  n  wise  men,  wi, ,  wn,  sequentially  announces  that  he  does  not  know 
whether  or  not  he  has  a  white  spot  on  his  forehead,  the  question  is  what  would 
the  last  wise  man  wn+\  report. 

For  all  n  >  1,  it  turns  out  that  the  last — (n  +  1)^ — wise  man  knows  he  is 
marked.  The  case  of  two  wise  men  is  simple.  The  reasoning  runs  essentially  by 
contradiction.  The  second  wise  man  reasons  as  follows: 

Suppose  I  were  not  marked.  Then  W\  would  have  seen  this,  and  knowing 
that  at  least  one  of  us  is  marked,  he  would  have  inferred  that  he  was 
the  marked  one.  But  w\  has  expressed  ignorance;  therefore,  I  must  be 
marked. 

Consider  now  the  case  of  n  =  3  wise  men  wi, 11)2,11)3.  After  w  1  announces  that 
he  does  not  know  that  he  is  marked,  W2  and  W3  both  infer  that  at  least  one  of 
them  is  marked.  For  if  neither  W2  nor  W3  were  marked,  wi  would  have  seen  this 
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and  would  have  concluded — and  stated — that  he  was  the  marked  one,  since  he 
knows  that  at  least  one  of  the  three  is  marked.  At  this  point  the  puzzle  reduces 
to  the  two-men  case:  both  w2  and  w3  know  that  at  least  one  of  them  is  marked, 
and  then  w2  reports  that  he  does  not  know  whether  he  is  marked.  Hence  w3 
proceeds  to  reason  as  previously  that  he  is  marked. 

In  general,  consider  n  +  1  wise  men  wi, . . .  ,wn,wn+i  ,n  >  1.  After  the  first 
j  wise  men  wi, ... ,  Wj  have  announced  that  they  do  not  know  whether  they  are 
marked,  for  j  =  1 , ...  ,n,  the  remaining  wise  men  1, . . . ,  wn+i  infer  that  at 
least  one  of  them  is  marked.  This  holds  for  j  =  n  as  well,  which  means  that  the 
last  wise  man  wn+i  will  infer  (and  announce,  owing  to  his  honesty)  that  he  is 
marked. 

The  question  is  how  to  formalize  this  in  our  logic.  Again  consider  the  case 
of  two  wise  men  u>i  and  w2.  Let  M^,i  £  {1,2}  denote  the  proposition  that  Wi 
is  marked.  For  any  proposition  P,  we  will  write  A'j(P)  as  an  abbreviation  for 
KWi(P).  We  will  only  need  three  premises: 


Sr  =  C^K^M 1)) 

52  =  C(Mi  V  M2) 

53  =  C{pM2  ^K\^M2)) 


The  first  premise  says  that  it  is  common  knowledge  that  the  first  wise  man 
does  not  know  whether  he  is  marked.  Although  it  sounds  innocuous,  note  that 
a  couple  of  assumptions  are  necessary  to  obtain  this  premise  from  the  mere 
fact  that  w\  has  announced  his  ignorance.  First,  truthfulness — we  must  assume 
that  the  wise  men  do  not  lie,  and  further,  that  each  one  of  them  knows  that 
they  are  all  truthful.  And  second,  each  wise  man  must  know  that  the  other 
wise  men  will  hear  the  announcement  and  believe  it.  Premise  S2  says  that  it  is 
common  knowledge  that  at  least  one  of  the  wise  men  is  marked.  Observe  that 
the  announcement  by  the  king  is  crucial  for  this  premise  to  be  justified.  The 
two  wise  men  can  see  each  other  and  thus  they  individually  know  Mi  V  M2. 
However,  each  of  them  may  not  know  that  the  other  wise  man  knows  that  at 
least  one  of  them  is  marked.  For  instance,  w\  may  believe  that  he  is  not  marked, 
and  even  though  he  sees  that  w2  is  marked,  he  may  believe  that  w2  does  not 
know  that  at  least  one  of  them  is  marked,  as  w2  cannot  see  himself.  Finally, 
premise  S3  states  that  it  is  common  knowledge  that  if  w2  is  not  marked,  then 
u’i  will  know  it  (because  w\  can  see  w2).  From  these  three  premises  we  are  to 
conclude  that  it  is  common  knowledge  that  w2  is  marked.  Symbolically,  we  need 
to  derive  the  judgment  {Si,  S2,  S3}  b  C(M2).  If  we  have  encoded  the  epistemic 
propositional  logic  in  a  predicate  calculus,  then  we  can  achieve  this  immediately 
by  instantiating  Lemma  7  below  with  a  h  »i,  P  h  Mi  and  Q  1— »  M2 — without 
performing  any  inference  whatsoever.  This  is  what  we  have  done  in  Athena. 
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For  the  case  of  n  =  3  wise  men  our  set  of  premises  will  be: 

51  =  C(-’K1(M1)) 

52  =  C( Ml  V  M2  V  m3) 

s3  =  c(-<(m2  v  m3)  v  m3))) 

S4  =  C(^I<2(M2)) 

53  =  C(~iM3  =>  K2(-*M3)) 

Consider  now  the  general  case  of  n  +  1  wise  men  wi, . . . ,  wn,  wn+i-  For  any 
i  =  1 , . . .  ,n,  define 

=  C^KtiMi)) 

Sl2  =  C(Mi  V  ■  ■  •  V  M„+1) 

Si  =  C(-(Mi+1  V  •  •  •  V  M„+1)  =»  Ki^Mi+i  V  •  •  •  V  Mn+1))) 

The  set  of  premises,  which  we  will  denote  by  J7n+i,  can  now  be  defined  as 

n 

<4+i  =  {C(M1  V  •  •  •  V  Mn+1)}  y  (Si,  S‘3} 

i=l 

Hence  f?n+1  has  a  total  of  2n  +  1  elements.  Note  that  S2  is  the  commonly 
known  disjunction  Af4  V  •  •  ■  V  Mn+i  and  a  known  premise,  i.e. ,  a  member  of 
Qn+i-  However,  S2  for  i  >  1  is  not  a  premise.  Rather,  it  becomes  derivable 
after  the  ft1  wise  man  has  made  his  announcement.  Managing  the  derivation 
of  these  propositions  and  eliminating  them  via  applications  of  the  cut  is  the 
central  function  of  the  algorithm  below.  Before  we  present  the  algorithm  we 
state  a  couple  of  key  lemmas. 

Lemma  6.  Consider  any  agent  a  and  propositions  P,Q,  and  let  R\,R2,R3  be 
the  following  three  propositions: 

1.  Ri  =  -'Ka(P); 

2.  R2  =  KahQ  =*•  P); 

3.  Rs  =  => Ka(-rQ) 

Then  {R\  A  R2  A  R3}  h  Q. 

Proof.  By  the  following  sequent  derivation: 

1.  {ki  A  f?2  A  JJs}  h  JJi  [Reflex],  A-Ei 

2.  { S  A  A 2  A  S3  }  \  R/2  [Reflex],  A- E\ ,  A-  R> 

3.  {Si  A  f?2  A  S3}  h  R3  [Reflex],  A-E2 

4.  {Si  A  S2  A  S3}  h  Ka(-iQ)  =>  Ka(P)  2,  [K],  =>-E 

5.  {Si  A  S2  A  S3}  I — 'Q  =>■  Ka(P)  3,  4,  Lemma  2 

6.  {Si  A  S2  A  S3}  I — 'Ka(P)  =>■  -i-iQ  5,  Lemma  3 

7.  {Si  A  S2  A  S3}  I — '-"C  6,  1,  =>-E 

8.  {Si  A  S2  A  S3}  KQ  7,  [-i-S] 

□ 
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Lemma  7.  Consider  any  agent  a  and  propositions  P,Q.  Define  R\  and  R3 
as  in  Lemma  6 ,  let  R2  =  PVQ,  and  let  Si  =  C(Ri)  for  i  =  1,2,3.  Then 

{.S,s2..s;3}i-c.m 


Proof.  Let  R'2  =  ~>Q  =>■  P  and  consider  the  following  derivation: 


1.  {Si,  &,  S3}  h  Si 

2.  (Si,  S2,  S3}  hS2 

3.  {Si,  So,  S3}  h  S3 

4.  0  b  (Pvg)^(oQ^P) 

5.  (Si,S2,S3}l-C'((Pvg)=S.'(-.Q=>P)) 

6.  {Si,S2,S3}bC(PvQ)=>C(-.Q=s>P) 

7.  {Si,S2,S3}bC(-.Q=>P) 

8.  {Si,S2,S3}  \-C(-oQ=>P)=>C(KahQ=>P)) 

9.  {Si,S2,S3}  hC(KahQ^P)) 

10.  {Pi  A  Ka{~oQ  =4>  P)  A  P3}  b  Q 

11.  0  b  (Pi  A  Ka{~oQ  =>  P)  A  P3)  =>  Q 

12.  {Si,  S2,  S3}  b  C((Pi  A  Ka{~oQ  =>P)  A  P3)  =>  Q) 

13.  {Si,S2,S3}  bC(Pi  AKahQ^P)  ARs)^C(Q) 

14.  {Si,S2,S3}  hC(Pi  AA'a(oQ^P)  A  P3) 

15.  {Si,S2,S3}  bC(Q) 


[  Rejlefi 
[Reflex] 

[Reflex] 

Lemma  4a 

4,  [C-I] 

5,  [CK],  [ =>-E] 

6,  2,  [=>-E] 

[p] 

8,  7,  [=>-P] 

Lemma  6 
10,  [=>-/] 

11,  [C-I] 

12,  [Ck],  [^-E] 

1,  3,  9,  Lemma  5,  [A-/] 

13,  14,  [=>-£] 


□ 


Our  method  can  now  be  stated  as  follows: 

#^{S11,S21,S31}; 

E<-$  bSf; 

Use  Lemma  7  to  derive 

If  n  =  1  halt 

else 

For  i  =  2  to  n  do 

begin 

£'<-  {S{,Sj,Sj}  bS*+1; 

Use  Lemma  7  to  derive  17'; 
!7"^<2>bS^+1; 

Use  the  cut  on  E  and  E'  to  derive  E"  \ 
E<—  E" 

end 


The  loop  variable  i  ranges  over  the  interval  2, . . . ,  n.  For  any  i  in  that  interval, 
we  write  F1  and  El  for  the  values  of  F  and  E  upon  conclusion  of  the  i^1  iteration 
of  the  loop.  A  straightforward  induction  on  i  will  establish: 

Lemma  8  (Algorithm  correctness).  For  any  i  6  {2,..., n}, 


&  =  {C(Mi  V  •  •  •  VMn+1)}  U  {Sf,S|} 

3= 1 


while  El  =  <2>!  b  S*+1. 
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It  follows  that  <f>n  =  f2n+1  and  En  =  <2>n  h  S%+1  =  Qn+1  h  +1I2„+i  h  C(Mn+1) 
which  is  our  goal. 

It  is  noteworthy  that  no  such  correctness  argument  is  necessary  in  the  formu¬ 
lation  of  the  algorithm  as  an  Athena  method.  Athena  methods  are  guaranteed 
to  be  sound.  As  long  as  the  produced  result  is  of  the  right  form  (in  our  case,  a 
sequent  of  the  form  f2n+1  b  C(Mn+i)),  we  can  be  assured  that  the  result  follows 
logically  from  the  contents  of  the  assumption  base. 

4  Athena  implementation 

In  this  section  we  present  the  Athena  encoding  of  the  epistemic  logic  and  our 
method  for  solving  the  generalized  version  of  the  wise  men  puzzle  (refer  to  the 
Appendix  for  a  brief  review  of  Athena).  We  begin  by  introducing  an  uninter¬ 
preted  domain  of  epistemic  agents:  (domain  Agent).  Next  we  represent  the  ab¬ 
stract  syntax  of  the  propositions  of  the  logic.  The  following  Athena  datatype 
mirrors  the  abstract  grammar  for  propositions  that  was  given  in  the  beginning 
of  Section  2: 

(datatype  Prop 
True 
False 

(Atom  Boolean) 

(Not  Prop) 

(And  Prop  Prop) 

(Or  Prop  Prop) 

(If  Prop  Prop) 

(Knows  Agent  Prop) 

(Common  Prop) ) 

We  proceed  to  introduce  a  binary  relation  sequent  that  may  obtain  between 
a  finite  set  of  propositions  and  a  single  proposition: 

(declare  sequent  (->  ((FSet-Of  Prop)  Prop)  Boolean)) 

Here  FSet-Of  is  a  unary  sort  constructor:  for  any  sort  T,  (FSet-Of  T)  is  a  new 
sort  representing  the  set  of  all  finite  sets  of  elements  of  T.  Finite  sets  are  built 
with  two  polymorphic  constructors:  the  constant  null,  representing  the  empty 
set;  and  the  binary  constructor  Insert,  which  takes  an  element  x  of  sort  T  and 
a  finite  set  S  (of  sort  (FSet-Of  T) )  and  returns  the  set  {x}  U  S.  We  also  have  all 
the  usual  set-theoretic  operations  available  (union,  intersection,  etc.). 

The  intended  interpretation  is  that  if  (sequent  S  P )  holds  for  a  set  of  propo¬ 
sitions  S  and  a  proposition  P,  then  the  sequent  S  h  P  is  derivable  in  the  epis¬ 
temic  logic  via  the  rules  presented  in  Section  2.  Accordingly,  we  introduce  axioms 
capturing  those  rules.  For  instance,  the  conjunction  introduction  rule  is  repre¬ 
sented  by  the  following  axiom: 

(define  And-I 

(forall  ?B  ?P  ?Q 

(if  (and  (sequent  ?B  ?P) 

(sequent  ?B  ?Q)) 

(sequent  ?B  (And  ?P  ?Q))))) 
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Note  that  the  lowercase  and  above  is  Athena’s  built-in  conjunction  operator,  and 
hence  represents  conjunction  at  the  metalanguage  level,  whereas  And  represents 
the  object-level  conjunction  operator  of  the  epistemic  logic. 

The  cut  rule  and  the  common  knowledge  introduction  (necessitation)  rule 
become: 

(define  cut 

(forall  ?B1  ?B2  ?P  ?Q 

(if  (and  (sequent  ?B1  ?P) 

(sequent  (insert  ?P  ?B2)  ?Q)) 

(sequent  (union  ?B1  ?B2)  ?Q)))) 

(define  common-intro-axiom 
(forall  ?P  ?B 

(if  (sequent  null  ?P) 

(sequent  ?B  (Common  ?P))))) 

The  remaining  rules  are  encoded  by  similar  first-order  axioms. 

We  next  proceed  to  derive  several  lemmas  that  are  useful  for  the  proof. 
Some  of  these  lemmas  are  derived  completely  automatically  via  the  ATPs  that 
are  integrated  with  Athena.  For  instance,  the  cut  rule  is  proved  automatically 
(in  about  10  seconds).  As  another  example,  the  following  result — part  (b)  of 
Lemma  4 — is  proved  automatically: 

(forall  ?B  ?P1  ?P2 

(if  (and  (sequent  null  (If  ?P1  ?P2)) 

(sequent  ?B  (Common  ?P1))) 

(sequent  ?B  (Common  ?P2)))) 

Other  lemmas  are  established  by  giving  natural  deduction  proofs.  For  instance, 
the  proof  of  Lemma  6  in  Section  3  is  transcribed  virtually  verbatim  in  Athena, 
and  validated  in  a  fraction  of  a  second.  (The  fact  that  the  proof  is  abridged — 
i.e.,  multiple  steps  are  compressed  into  single  steps — is  readily  handled  by  in¬ 
voking  ATPs  that  automatically  fill  in  any  gaps.)  Finally,  we  are  able  to  prove 
Lemma  7,  which  is  the  key  technical  lemma.  Utilizing  the  higher-order  charac¬ 
ter  of  our  encoding,  we  then  define  a  method  main-lemma  that  takes  an  arbi¬ 
trary  list  of  agents  [ai  ■  ■  ■  an\ ,  n  >  1,  and  specializes  Lemma  7  with  P  i— >  Mai , 
Q  i — *  M02  V  •••  VMan,  and  a  i— >  a\  (recall  that  for  any  agent  a,  Ma  signi¬ 
fies  that  a  is  marked).  So,  for  instance,  the  application  of  main-lemma  to  the 
list  [ai,  <12,123]  would  derive  the  conclusion  {Si,  S2,  S3}  b  C(Ma2  V  M<,3),  where 
Si  =  C(-^Kai(Mai)),  S2  =  C(Mai  V  Ma2  V  Ma3 ) ,  and 

S3  =  CHM02  V  Mas)  =>  Kai(-,(Ma2  V  M„3))) 

We  also  need  a  simple  result  shuffle  asserting  the  equality  F,  Pi,P-2  =  F,  P2,  Pi 
(i.e.,  F  U  {Pi}  U  {P2}  =FU  {P2}  U  {Pi}). 

Using  these  building  blocks,  we  express  the  tactic  for  solving  the  generalized 
wise  men  problem  as  the  Athena  method  solve  below.  It  takes  as  input  a  list  of 
agents  representing  wise  men,  with  at  least  two  elements.  Note  that  the  for  loop 
in  the  pseudocode  algorithm  has  been  replaced  by  recursion. 
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(define  (solve  wise-men) 

(dletrec 

((loop  (method  (wise-men  th) 

(dmatch  wise-men 
( [_]  ( ! claim  th) ) 

((list-of  _  rest) 

(diet  ((new-th  (! main-lemma  wise-men))) 

(dmatch  [th  new-th] 

([(sequent  context  Q2) 

(sequent  (insert  Q1 

(insert  Q2  (insert  Q3  null)))  P)] 

(diet  ((cut-th 

(! derive  (sequent 
(union 
context 

(insert  Q1  (insert  Q3  null))) 

P) 

[th  new-th  shuffle  cut]))) 

(Hoop  rest  cut-th)))))))))) 

(diet  ((init  ( ! prove-goal-2  wise-men))) 

(lloop  (tail  wise-men)  init)))) 

Assuming  that  wl,  w2,  w3  are  agents  representing  wise  men,  invoking  the  method 
solve  with  the  list  [wl  w2  w3] )  as  the  argument  will  derive  the  appropriate  result: 
i?3  b  (Common  (isMarked  w3)),  where  i?3  is  the  set  of  premises  for  the  three- men 
case,  as  defined  in  the  previous  section. 

5  Related  work 

The  wise  men  problem  became  a  staple  of  epistemic  AI  literature  after  being 
introduced  by  McCarthy  [30].  Formalizations  and  solutions  of  the  two- wise- men 
problem  are  found  in  a  number  of  sources  [26, 40, 19] ,  most  of  them  in  simple 
multi- agent  epistemic  logics  (without  common  knowledge).  Several  variations 
have  been  given;  e.g.,  Konolige  has  a  version  in  which  the  third  wise  man  states 
that  he  does  not  know  whether  he  is  marked,  but  that  he  would  know  if  only  the 
second  wise  man  were  wiser  [28].  Ballim  and  Wilks  [8]  solve  the  three-men  ver¬ 
sion  of  the  puzzle  using  the  “nested  viewpoints”  framework.  Vincenzo  Pallotta’s 
solution  [33]  is  similar  but  his  ViewGen  framework  facilitates  agent  simulation. 
Kim  and  Kowalski  [27]  use  a  Prolog-based  implementation  of  metareasoning  to 
solve  the  same  version  of  the  problem  using  common  knowledge.  A  more  natural 
proof  was  given  by  Aiello  et  al.  [1]  in  a  rewriting  framework. 

The  importance  of  metareasoning  and  metaknowledge  for  intelligent  agents  is 
extensively  discussed  in  “Logical  foundations  of  Artifical  Intelligence”  by  Gene- 
sereth  and  Nillson  [19]  (it  is  the  subject  of  an  entire  chapter).  They  stress  that  the 
main  advantage  of  an  explicit  encoding  of  the  reasoning  process  is  that  it  makes 
it  possible  to  “create  agents  capable  of  reasoning  in  detail  about  the  inferential 
abilities  of  and  beliefs  of  other  agents,”  as  well  as  enabling  introspection. 
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The  only  work  we  are  aware  of  that  has  an  explicit  encoding  of  an  epistemic 
logic  in  a  rich  metalanguage  is  a  recent  project  [29]  that  uses  the  Calculus  of 
Constructions  (Coq  [11]).  However,  there  are  important  differences.  First,  they 
encode  a  Hilbert  proof  system,  which  has  an  adverse  impact  on  the  readability 
and  writability  of  proofs.  The  second  and  most  important  difference  is  our  em¬ 
phasis  on  reasoning  efficiency.  The  seamless  integration  of  Athena  with  state-of- 
the-art  provers  such  as  Vampire  and  Spass  is  crucial  for  automation,  as  it  enables 
the  user  to  skip  tedious  steps  and  keep  the  reasoning  at  a  high  level  of  detail. 
Another  distinguishing  aspect  of  our  work  is  our  reliance  on  tactics.  Athena  uses 
a  block-structured  natural-deduction  style  not  only  for  writing  proofs  but  also 
for  writing  proof  tactics  (“methods”).  Proof  methods  are  much  easier  to  write  in 
this  style,  and  play  a  key  role  in  proof  automation.  Our  emphasis  on  automation 
also  differentiates  our  work  from  that  of  Basin  et  al.  [9]  using  Isabelle,  which  only 
addresses  proof  presentation  in  modal  logics,  not  automatic  proof  discovery. 
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A  Athena  Overview 

Athena  is  a  new  interactive  theorem  proving  system  that  incorporates  facilities 
for  model  generation,  automated  theorem  proving,  and  structured  proof  repre¬ 
sentation  and  checking.  It  also  provides  a  higher-order  functional  programming 
language,  and  a  proof  abstraction  mechanism  for  expressing  arbitrarily  compli¬ 
cated  inference  methods  in  a  way  that  guarantees  soundness,  akin  to  the  tactics 
and  tacticals  of  LCF-style  systems  such  as  HOL  [20]  and  Isabelle  [34].  Proof  au¬ 
tomation  is  achieved  in  two  ways:  first,  through  user-formulated  proof  methods; 
and  second,  through  the  seamless  integration  of  state-of-the-art  ATPs  such  as 
Vampire  [41]  and  Spass  [42]  as  primitive  black  boxes  for  general  reasoning.  For 
model  generation,  Athena  integrates  Paradox  [10],  a  new  highly  efficient  model 
finder.  For  proof  representation  and  checking,  Athena  uses  a  block-structured 
Fitch-style  natural  deduction  calculus  [35]  with  novel  syntactic  constructs  and  a 
formal  semantics  based  on  the  abstraction  of  assumption  bases  [3].  Most  inter¬ 
estingly,  a  block-structured  natural  deduction  format  is  used  not  only  for  writing 
proofs,  but  also  for  writing  tactics  (methods).  This  is  a  novel  feature  of  Athena; 
all  other  tactic  languages  we  are  aware  of  are  based  on  sequent  calculi.  Tactics 
in  this  style  are  considerably  easier  to  write  and  remarkably  useful  in  making 
proofs  more  modular  and  abstract. 

Athena  has  been  used  to  implement  a  proof-emitting  optimizing  compiler 
[36];  to  integrate  model  checking  and  theorem  proving  for  relational  reasoning 
[4] ;  to  implement  various  “certifying”  algorithms  [5] ;  to  verify  the  core  operations 
of  a  Unix-like  file  system  [6];  to  prove  the  correctness  of  dataflow  analyses  [22]; 
and  to  reason  about  generic  software  [32].  This  section  presents  parts  of  Athena 
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relevant  to  understanding  the  code  in  this  paper.  A  more  thorough  presentation 
of  Athena’s  syntax  and  semantics  can  be  found  elsewhere  [7]. 

In  Athena,  an  arbitrary  universe  of  discourse  (sort)  is  introduced  with  a 
domain  declaration,  for  example: 

(domain  Real) 

(domain  Person) 

Function  symbols  and  constants  can  then  be  declared  on  the  domains,  e.g.: 

(declare  +  (->  (Real  Real)  Real)) 

(declare  joe  Person) 

Relations  are  functions  whose  range  is  the  predefined  sort  Boolean,  e.g., 

(declare  <  (->  (Real  Real)  Boolean)) 

Inductively  generated  domains  are  introduced  as  datatypes,  e.g., 

(datatype  Nat 
zero 

(succ  Nat)) 

Here  Nat  is  freely  generated  by  the  constructors  zero  and  succ.  This  is  equivalent 
to  issuing  the  declarations  (domain  Nat),  (declare  zero  Nat), 

(declare  succ  (->  (Nat)  Nat)) 

and  additionally  postulating  a  number  of  axioms,  as  well  as  an  appropriate 
induction  principle,  that  constrain  Nat  to  be  freely  generated  by  zero  and  succ. 
The  axioms  and  the  induction  principle  are  automatically  generated  when  the 
user  defines  the  datatype. 

The  basic  data  values  in  Athena  are  terms  and  propositions.  Terms  are  s- 
expressions  built  from  declared  function  symbols  such  as  +  and  pi ,  and  from  vari- 
ables,  written  as  ?/  for  any  identifier  I.  Thus  ?x,  (+  ?foo  pi),  (+  (+  ?x  ?y)  ?z), 
are  all  terms.  The  (most  general)  sort  of  a  term  is  inferred  automatically;  the 
user  does  not  have  to  annotate  variables  with  their  sorts.  A  proposition  P  is 
either  a  term  of  sort  Boolean  (say,  (<  pi  (+  ?x  ?y)));  or  an  expression  of  the 
form  (not  P)  or  (0  Pi  P2)  for  ©  G  {and,  or,  if ,  iff };  or  (Q  xi  ■  ■  ■  xn  P)  where 
Q  G  {f orall,  exists}  and  each  Xi  a  variable.  Athena  also  checks  the  sorts  of 
propositions  automatically  using  a  Hindley-Milner-like  type  inference  algorithm. 

The  user  interacts  with  Athena  via  a  read-eval-print  loop.  Athena  displays 
a  prompt  >,  the  user  enters  some  input  (either  a  phrase  to  be  evaluated  or  a 
top-level  directive  such  as  define,  assert,  declare,  etc.),  Athena  processes  the 
user’s  input,  displays  the  result,  and  the  loop  starts  anew. 

The  most  fundamental  concept  in  Athena  is  the  assumption  base — a  finite  set 
of  propositions  that  are  assumed  to  hold,  representing  our  “axiom  set”  or  “knowl¬ 
edge  base” .  Athena  starts  out  with  the  empty  assumption  base,  which  then  gets 
incrementally  augmented  with  the  conclusions  of  the  deductions  that  the  user 
successfully  evaluates  at  the  top  level  of  the  read-eval-print  loop,  proposition 
can  also  be  explicitly  added  into  the  global  assumption  base  with  the  top-level 
directive  assert. 
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An  Athena  deduction  D  is  always  evaluated  in  a  given  assumption  base  j5 — 
a  finite  set  of  propositions  that  are  assumed  to  hold  for  the  purposes  of  D. 
Evaluating  D  in  (3  will  either  produce  a  proposition  P  (the  “conclusion”  of  D  in 
/?),  or  else  it  will  generate  an  error  or  will  diverge.  If  D  does  produce  a  conclusion 
P,  Athena’s  semantics  guarantee  f3  |=  P,  i.e.,  that  P  is  a  logical  consequence  of 
(3.  There  are  several  syntactic  forms  that  can  be  used  for  deductions. 

The  form  pick-any  introduces  universal  generalizations:  (pick-any  h  ■  ■  ■  I„  D) 
binds  the  names  I\  ■  ■  ■  In  to  fresh  variables  Vi, . . .  ,v„  and  evaluates  D.  If  D  yields 
a  conclusion  P,  the  result  returned  by  the  entire  pick-any  is  (Vvi, . . .  ,vn)  P. 

The  form  assume  introduces  conditionals:  to  evaluate  (assume  P  D)  in  an 
assumption  base  (3,  we  evaluate  D  in  j3  U  {P}.  If  that  produces  a  conclusion  Q , 
the  conditional  P  =k  Q  is  returned  as  the  result  of  the  entire  assume.  The  form 
(assume-let  ((/  P))  D )  works  like  assume,  but  also  lexically  binds  the  name  I 
to  the  hypothesis  P  within  D. 

The  form  (diet  ((/i  Dp  ■■■  (/„  Dn ))  D)  is  used  for  sequencing  and  nam¬ 
ing  deductions.  To  evaluate  such  a  deduction  in  j3,  we  first  evaluate  Di  in  (3  to 
obtain  a  conclusion  P\.  We  then  bind  I\  to  P\,  insert  P\  into  (3,  and  continue 
with  D2-  The  conclusions  Pi  of  the  various  Di  are  thus  incrementally  added 
to  the  assumption  base,  becoming  available  as  lemmas  for  subsequent  use.  The 
body  D  is  then  evaluated  in  j3  U  {Pi, . . . ,  Pn},  and  its  conclusion  becomes  the 
conclusion  of  the  entire  diet. 
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Abstract 

We  describe  our  approach  to  building  advanced  synthetic 
characters,  within  the  paradigm  of  logic-based  AI.  Such  char¬ 
acters  don’t  merely  evoke  beliefs  that  they  have  various  men¬ 
tal  properties;  rather,  they  must  actually  have  such  properties. 
You  might  (e.g.)  believe  a  standard  synthetic  character  to  be 
evil,  but  you  would  of  course  be  wrong.  An  advanced  syn¬ 
thetic  character,  however,  can  literally  be  evil,  because  it  has 
the  requisite  desires,  beliefs,  and  cognitive  powers.  Our  ap¬ 
proach  is  based  on  our  RASCALS  architecture,  which  uses 
simple  logical  systems  (first-order  ones)  for  low-level  (per¬ 
ception  &  action)  and  mid-level  cognition,  and  advanced  log¬ 
ical  systems  (e.g.,  epistemic  and  deontic  logics)  for  more  ab¬ 
stract  cognition.  To  focus  our  approach  herein,  we  provide  a 
glimpse  of  our  attempt  to  bring  to  life  one  particular  advanced 
synthetic  character  from  the  “dark  side”  —  the  evil  charac¬ 
ter  known  simply  as  E.  Building  E  entails  that,  among  other 
things,  we  formulate  an  underlying  logico-mathematical  def¬ 
inition  of  evil,  and  that  we  manage  to  engineer  both  an  ap¬ 
propriate  presentation  of  E,  and  communication  between  E 
and  humans.  For  presentation,  which  we  only  encapsulate 
here,  we  use  several  techniques,  including  muscle  simula¬ 
tion  in  graphics  hardware  and  approximation  of  subsurface 
scattering.  For  communication,  we  use  our  own  new  “proof- 
based”  approach  to  Natural  Language  Generation  (NLG).  We 
provide  an  account  of  this  approach. 


The  Dearth  of  AI  in  AI 

There’s  an  unkind  joke  —  which  made  the  rounds  (e.g.)  at 
the  Fall  2004  AAAI  Fall  Symposium  on  Human-Level  AI  — 
about  the  need  to  create,  within  AI,  a  special  interest  group 
called  ‘AI’.  This  kind  of  cynicism  springs  from  the  not  un¬ 
common,  and  not  totally  inaccurate,  perception  that  most  of 
AI  research  is  aimed  at  exceedingly  narrow  problems  light 
years  away  from  the  cognitive  capacities  that  distinguish  hu¬ 
man  persons.1 


‘The  R&D  described  in  this  paper  has  been  supported  in  part 
by  much  appreciated  grants  from  AFRL-Rome  and  DARPA-IPTO. 

1  An  endless  source  of  confirming  examples  can  be  found  in  the 
pages  of  the  Machine  Learning  journal.  The  dominant  learning 
technique  that  you  yourself  employ  in  striving  to  learn  is  reading', 
witness  what  you're  doing  at  the  moment.  Yet,  a  vanishingly  small 
amount  of  R&D  on  learning  is  devoted  to  getting  a  computer  pro¬ 
gram  to  learn  by  reading. 


Human-level  AI  is  now  so  unusual  that  an  entire  upcom¬ 
ing  issue  of  AI  Magazine  will  be  devoted  to  the  subject  — 
a  bit  odd,  given  that,  at  least  when  the  field  was  young, 
AI's  journal  of  record  would  have  routinely  carried  papers 
on  mechanizing  aspects  of  human-level  cognition.  Seminal 
AI  thinkers  like  Simon,  Newell,  Turing  —  these  researchers 
didn't  shy  away  from  fighting  to  capture  human-level  intelli¬ 
gence  in  machine  terms.  But  now  their  attitude  seems  mori¬ 
bund. 

But  gaming,  simulation,  and  digital  entertainment  (and 
hereafter  we  refer  simply  to  ‘gaming’  to  cover  this  entire 
field/market),  thankfully,  are  different:  ultimately  anyway, 
they  call  for  at  least  the  appearance  of  human-level  AI 
(Bringsjord  2001 ).  (On  a  case-by-case  basis,  as  various 
games  show  (e.g..  The  Sims  (Electronic  Arts  Inc.  2000)),  a 
non- advanced  character  will  of  course  do  just  fine.)  Gaming 
doesn't  strive  just  for  a  better  SAT-based  planner,  or  another 
tweak  in  a  learning  algorithm  that  doesn't  relate  in  the  least 
to  human  learning.  A  SAT  planner  doesn’t  constitute  a  vir¬ 
tual  person.  But  that’s  precisely  what  we  want  in  gaming,  at 
least  ultimately.  And  even  in  the  short  term  we  want  char¬ 
acters  that  at  least  seem  human.  Methodologically  speaking, 
gaming’s  best  bet  for  characters  that  seem  human  is  to  bite 
the  bullet  and  strive  to  engineer  characters  that  have  what  it 
takes  to  be  human.  This,  at  least,  is  our  strategy. 

Gaming  and  Full-Blown  Personhood 

Now,  there  are  various  ways  to  get  clearer  about  what  gam¬ 
ing.  at  least  in  the  long-term,  needs  when  it  comes  to  human- 
level  intelligence.  One  way  is  to  say  simply  that  gaming 
needs  artificial  creatures  which,  behaviorally  at  any  rate,  sat¬ 
isfy  one  or  more  plausible  proposed  definitions  of  person- 
hood  in  the  literature.  One  such  definition  has  been  pro¬ 
posed  by  Bringsjord  in  (Bringsjord  1997).  This  definition 
essentially  amounts  to  the  view  that  a:  is  a  person  if  and  only 
if  x  has  the  capacity 

1 .  to  “will,”  to  make  choices  and  decisions,  set  plans  and  projects 
—  autonomously; 

2.  for  consciousness,  for  experiencing  pain  and  sorrow  and  happi¬ 
ness,  and  a  thousand  other  emotions  —  love,  passion,  gratitude, 
and  so  on; 

3.  for  xe//-consciousness,  for  being  aware  of  his/her  states  of  mind, 
inclinations,  preferences,  etc.,  and  for  grasping  the  concept  of 
him/herself; 
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4.  to  communicate  through  a  language; 

5.  to  know  things  and  believe  things,  and  to  believe  things  about 
what  others  believe,  and  to  believe  things  about  what  others  be¬ 
lieve  about  one’s  beliefs  (and  so  on); 

6.  to  desire  not  only  particular  objects  and  events,  but  also  changes 
in  his  or  her  character; 

7.  to  reason  (for  example,  in  the  fashion  exhibited  in  the  writing 
and  reading  of  this  very  paper). 

Unfortunately,  this  list  is  daunting,  especially  if,  like  us, 
you  really  and  truly  want  to  engineer  a  virtual  person  in 
the  short  term.  A  large  part  of  the  problem  is  conscious¬ 
ness,  which  we  still  don’t  know  how  to  represent  in  third- 
person  machine  terms  (Bringsjord  1998;  Bringsjord  2001). 
But  even  if  we  leave  aside  consciousness,  the  rest  of  the 
attributes  in  the  above  list  make  for  mighty  tough  chal¬ 
lenges.  In  the  section  “Making  the  Challenge  of  Person- 
hood  Tractable"  we  shall  retreat  from  this  list  to  someting 
doable  in  the  near  term,  guided  by  particular  scenarios  that 
make  natural  homes  for  E.  But  in  the  end,  whatever  appears 
on  this  list  is  an  engineering  target  for  us;  in  the  long  term 
we  must  confront  each  clause.  Accordingly,  in  the  section 
“How  Does  E  Talk?"  we  explain  how  we  are  shooting  for 
clause  4,  communication.  We  have  made  progress  on  some 
of  the  other  clauses,  but  there  is  insufficient  space  to  present 
that  progress  herein.  Clause  5  is  one  we  believe  we  have 
pretty  much  satisfied,  via  the  formalization  and  implemen¬ 
tation  given  in  (Arkoudas  &  Bringsjord  2005).2 


putting  the  overall  problem  infecting  todays’s  virtual  char¬ 
acters,  all  of  the  cognitive  capacities  that  distinguish  human 
persons,  according  to  the  science  of  cognition  (e.g.,  (Gold¬ 
stein  2005)),  are  missing.  Even  the  state  of  the  art  using  cog¬ 
nitive  architectures  (e.g.,  SOAR)  is  primitive  when  stacked 
against  full-blown  personhood  (Ritter  et  al.  June  2002). 


Figure  1 :  Sample  Synthetic  Characters 


Current  State  of  the  Art  versus  Computational 
Persons 

Synthetic  Characters  in  Gaming 

What’s  being  done  now  in  gaming,  relative  to  full-blown 
personhood,  is  clearly  inadequate;  this  can  be  quickly  seen 
by  turning  to  some  standard  work:  Figure  1  shows  an  array 
of  synthetic  characters  from  the  gaming  domain;  these  will 
be  familiar  to  many  readers.3 

None  of  these  creatures  has  anything  close  to  the  distin¬ 
guishing  features  of  personhood.  Sustained  treatments  of 
synthetic  characters  and  how  to  build  them  are  similarly  lim¬ 
ited.  For  example,  consider  Figure  2,  taken  from  (Cham- 
pandard  2003).4  As  a  mere  FSA,  there  is  no  knowledge  and 
belief,  no  reasoning,  no  declarative  memories,  and  no  lin¬ 
guistic  capacity.  In  short,  and  this  is  perhaps  a  better  way  of 

2  A  preprint  is  available  online  at 
http://kryten.mrn.rpi.edu/arkoudas.bringsjord.clima.crc.pdf. 

3Worst  to  best,  in  our  eyes:  Top-left,  The  Legend  of  Zelda;  SC 
spits  text  upon  entering  room.  Top-right,  Chrono  Trigger;  tree¬ 
branching  conversations.  Middle-left,  Might  &  Magic  VI  (Shop¬ 
keepers).  Middle-right,  Superfly  Johnson  from  Daikatana;  behav¬ 
ior  scripting,  attempts  to  follow  player  and  act  as  a  sidekick  (fails!). 
Bottom-left,  Galatea  -  Interactive  Fiction  award  winner  for  Best 
NPC  of  2000  (text-based).  Bottom-right,  Sims  2.  But  even  here, 
nothing  like  what  our  RASCALS  architecture  has  is  present. 

4This  is  an  excellent  book,  and  it’s  used  in  our  lab  for  building 
synthetic  characters.  But  relative  to  the  loftier  goals  of  reaching 
bona  fide  personhood  in  artificial  characters,  there’s  clearly  a  lot  of 
work  to  be  done. 


What  About  Synthetic  Characters  in  Cutting  Edge 
Research? 

What  about  research-grade  work  on  synthetic  characters? 
Many  researchers  are  working  on  synthetic  characters,  and 
have  produced  some  truly  impressive  systems.  However, 
all  such  systems,  however  much  they  appear  to  be  human 
persons,  aren't.  We  now  consider  three  examples  of  such 
work,  and  show  in  each  that  the  character  architectures 
don't  have  the  underlying  cognitive  content  that  is  necessary 
for  personhood. 

REA 

An  agent  developed  by  (Cassell  et  al.  1999)  known  as  REA 
is  an  example  of  a  successful,  robust  agent  whose  developers 
focused  primarily  on  embodied  conversation  and  the  conver¬ 
sational  interface.  She  is  described  as  being  an  expert  in  the 
domain  of  real  estate,  and  interactions  with  REA  are  both 
believable  and  informative. 

REA.  however,  is  representative  of  many  of  the  indus¬ 
try’s  most  successful  agents  in  that  she  excels  at  content 
management,  but  fails  to  deliver  rich  emotive  and  cognitive 
functionality.  REA,  after  all,  cannot  generate  English  from 
arbitrary  underlying  knowledge.  Like  many  of  her  peers, 
REA’s  underlying  cognitive  capabilities  are  modeled  in 
an  ad-hoc  fashion.  Her  personality  is  in  no  way  defined; 
her  interactions  within  a  particular  situation  lack  subtlety 
and  depth.  While  she  excels  as  a  simulated  character  and 
a  conversational  agent,  she  is  bereft  of  the  rich  cognitive 
content  with  which  advanced  synthetic  characters  must 
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Moods 


Figure  2:  Impoverished  Formalism  for  Synthetic  Characters 


behave. 

The  BEAT  Architecture 

In  an  engaging  paper  by  (Gratch  etal.  2002),  an  architecture 
is  presented  for  developing  rich  synthetic  characters.  This 
architecture  is  known  as  the  Behavior  Expression  Anima¬ 
tion  Toolkit  Text-to-Nonverbal  Behavior  Module  (BEAT). 
Under  this  architecture,  emotion  and  cognitive  content  are 
produced  systematically  in  a  simulation-based  approach. 

Their  simulation-based  approach  is  built  on  top  of  ap¬ 
praisal  theories  of  emotion,  where  emotions  emerge  from 
analysis  of  events  and  objects  in  a  particular  domain  with 
respect  to  the  agent's  goals,  standards,  and  attitudes.  But  as 
Gratch  et  al.  themselves  point  out.  appraisal  theories  “are 
rather  vague  about  the  assessment  process.. .A  promising 
line  of  research  is  integrating  Al-based  planning  approaches, 
which  might  lead  to  a  concretization  of  such  theories."  We 
will  present  the  RASCALS  paradigm  as  one  that  utilizes  pre¬ 
cisely  the  Al-based  planning  techniques  Gratch  et  al.  regard 
as  promising. 

Unfortunately,  while  Gratch  et.  al  make  wonderful  ad¬ 
vancements  in  the  logistics  of  realizing  agents,  the  issue  of 
developing  rich  underlying  cognitive  content  is  eschewed. 
Even  assuming  that  their  simulation-based  approach  utilizes 
robust  Al-based  planning,  the  focus  is  not  on  developing 
true  cognitive  content  but  rather  on  its  simulation  and 
modeling. 

Believable  Interactive  Embodied  Agents 

An  approach  more  focused  on  building  believable  characters 
was  proposed  by  (Pelachaud  &  Poggi  2002).  They  argue  that 


research  should  include  three  distinct  phases: 

•  Phase  1:  Empirical  Research.  This  phase  involves  research 
“aimed  at  finding  out  the  regularities  in  the  mind  and  behavior 
of  Human  Agents,  and  at  constructing  models  of  them." 

•  Phase  2:  Modeling  Believable  Interactive  Embodied  Agents. 
Here,  "rules  are  formalized,  represented,  and  implemented  in 
the  construction  of  Agents.” 

•  Phase  3:  Evaluation.  Finally,  agents  are  tested  on  several  levels, 
including  “how  well  they  fit  the  User’s  needs  and  how  similar 
they  look  to  a  real  Human  Agent.” 

The  “rule  formalization”  characterized  in  Phase  2  is,  as 
Pelachaud  and  Poggi  point  out,  indispensable  when  building 
believable  characters.  Since  such  rule  formalizations  are  all 
expressible  in  first-order  logic,  their  approach  is  actually  a 
proper  subset  of  the  RASCALS  approach.  But  formalizing 
and  implementing  rules  is  not  enough  to  achieve  true  cog¬ 
nition;  after  all,  cognition  involves  much  more  than  simple 
rules/first-order  logic.  Iterated  beliefs  are  beyond  the  reach 
of  first-order  logic.  Finally,  while  Pelachaud  and  Poggi  elab¬ 
orate  on  linguistic  rules  and  formalizations,  they  fail  to  men¬ 
tion  anything  about  modeling  cognition  or  interacting  with  a 
given  knowledge  base,  and  they  make  no  remarks  concern¬ 
ing  the  logistics  behind  rule  formalization  and  implementa¬ 
tion.  The  agents  described  therein  all  possess  rudimentary 
cognitive  content  but  come  nowhere  close  to  true  cognitive 
or  emotive  capacity. 

Making  the  Challenge  of  Personhood 
Tractable 

How  can  we  make  the  challenge  of  engineering  a  virtual 
person  tractable  in  the  very  short  term?  Our  lab  has  a  two- 
part  answer.  First,  assimilate  everything  out  there  regarding 
the  craft  of  making  viewers  and  users  believe  that  the  syn¬ 
thetic  character  they  interact  with  is  a  genuine  person.  This 
is  the  same  route  that  was  followed  by  Bringsjord  and  Fer- 
rucci  in  the  design  of  the  BRUTUS  story  generation  system 
(Bringsjord  &  Ferrucci  2000).  In  a  nutshell,  B&F  studied 
the  literature  on  what  responses  are  desired  in  readers  by 
clever  authors,  and  then  reverse  engineered  back  from  these 
responses  to  a  story  generation  system  that  triggers  some  of 
them.  In  connection  with  synthetic  characters,  this  general 
strategy  has  impelled  us  to  build  up  a  large  library  on  the 
design  of  synthetic  charaters  in  stories  and  movies.  In  ad¬ 
dition,  we  have  built  up  a  library  of  characters  in  film  — 
specifically  one  that  specializes  in  candidates  for  true  evil. 
Within  the  space  we  have  herein,  however,  this  general  strat¬ 
egy,  and  the  results  so  far  obtained,  can’t  be  presented.  So 
we  will  settle  here  for  a  shortcut;  it’s  the  second  part  of  our 
two-part  answer.  The  shortcut  is  to  work  from  concrete  sce¬ 
narios  backwards  by  reverse  engineering.  We  currently  have 
two  detailed  scenarios  under  development.  One  is  based  on 
the  evil  people  whose  personalities  are  revealed  in  conversa¬ 
tions  in  (Peck  1983);  we  leave  this  one  aside  for  now.  The 
second  scenario,  which  is  part  of  R&D  undertaken  in  the 
area  of  wargaming,  can  be  summarized  as  follows.  (At  the 
conference,  we  would  provide  a  demo  of  conversation  with 
E  regarding  both  these  scenarios,  where  that  conversation 
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conforms  to  our  account  of  evil;  see  On  our  Formal  Account 
of  Evil.) 

E  in  Scenario  2,  and  Inference  Therefrom 

Let  us  imagine  a  man  named  simply  E.  a  brutal  warlord  in 
a  war-torn  country.  E  is  someone  you’re  going  to  have  to 
vanquish.  He  has  moved  up  the  ranks  of  the  underworld 
in  post-apocalyptic  America  after  “success”  in  many,  many 
murderous  missions.  E  has  taken  a  number  of  prisoners  from 
an  organization  (let’s  call  it  simply  O)  he  seeks  to  intimidate. 
O  is  chosen  specifically  because  it  is  trying  to  rebuild  the 
fractured  US  in  the  direction  of  a  new  federal  governing5. 
Conforming  to  what  has  unfortunately  become  a  gruesome 
pattern,  E  decides  to  film  the  beheading  of  one  of  these  poor 
prisoners,  and  to  release  the  video  to  O. 

Given  just  this  small  amount  of  information,  what  can  we 
infer  about  E’s  knowledge  and  reasoning?  That  it  has  at  least 
the  following  six  attributes: 

1.  Mixed  Representation.  E’s  knowledge  is  not  simply  linguistic 
or  symbolic  in  nature.  It  includes  visual  or  pictorial  knowledge 
as  well.  For  example,  E  clearly  is  thinking  in  terms  of  mental 
images,  because  he  plans  to  gain  leverage  from  the  release  of 
images  and  video.  In  addition,  though  it  isn’t  pleasant  to  con¬ 
template,  E  certainly  has  a  “mental  movie”  that  he  knows  he  can 
turn  into  real  life:  he  envisions  how  such  executions  work  before 
performing  them. 

2.  Tapestried.  Presumably  E’s  knowledge  of  his  prisoners  is  rel¬ 
atively  new.  But  this  new  knowledge  is  woven  together  with 
extensive  prior  knowledge  and  belief.  For  example,  in  E’s  case, 
he  has  extensive  knowledge  of  O.  and  its  principles  regarding 
treatment  of  prisoners. 

3.  Extreme  Expressivity.  E’s  knowledge  and  reasoning  requires 
highly  expressive  propositions.  For  example,  he  believes  that  O 
believes  that  it  is  universally  forbidden  to  execute  prisoners,  and 
he  believes  that  some  of  those  aiding  the  United  States’  rebuild¬ 
ing  effort  will  be  struck  with  fear  once  the  execution  is  complete 
and  suitably  publicized,  and  that  that  fear  will  affect  their  beliefs 
about  what  they  should  and  shouldn't  do. 

4.  Mixed  Inference  Types.  E’s  reasoning  is  based  not  only  on  de¬ 
ductive  inference,  but  also  on  educated  guesses  (abduction),  and 
probabilistic  inference  (induction). 

5.  Uses  Natural  Language.  E  communicates  in  natural  language, 
with  his  comrades,  and  with  others  as  well. 

6.  Multi-Agent  Reasoning.  E  is  of  course  working  in  coordinated 
fashion  with  a  number  of  accomplices,  and  to  be  effective,  they 
must  reason  well  as  a  group. 

Working  within  the  paradigm  of  logic-based  AI  (Bringsjord 
&  Ferrucci  1998a;  Bringsjord  &  Ferrucci  1998b;  Nilsson 
1991;  Genesereth  &  Nilsson  1987),  and  using  the  MARMML 
knowledge  representation  and  reasoning  system,  which  is 
based  on:  the  theory  known  as  mental  metalogic  (Yang 
&  Johnson-Laird  2000a;  Yang  &  Johnson-Laird  2000b; 
Yang  &  Bringsjord  2005;  Rinella,  Bringsjord,  &  Yang  2001; 
Yang  &  Bringsjord  2001a;  Yang  &  Bringsjord  2001b;  Yang, 
Braine,  &  O’Brien  1998),  the  Denotational  Proof  Language 


known  as  Athena  (Arkoudas  2000),  Barwisean  grids  for  di¬ 
agrammatic  knowledge  and  reasoning  (see  the  mathemati¬ 
cal  section  of  (Barwise  &  Etchemendy  1995)),  and  RAS- 
CALS6(see  Figure  3),  a  revolutionary  architecture  for  syn¬ 
thetic  characters,  we  are  building  a  virtual  version  of  E  that 
has  the  six  attributes  above. 


Figure  3:  RASCALS:  Rensselaer  Advanced  Synthetic 
Character  Architecture  for  Logical  Systems 


Brief  Remarks  on  the  RASCALS  Architecture 

Let  us  say  a  few  words  about  RASCALS,  a  brand  new  en¬ 
try  in  the  field  of  compuational  cognitive  modeling,  which 
revolves  around  what  are  called  cognitive  architectures 
(e.g.,  SOAR  (Rosenbloom,  Laird,  &  Newell  1993);  ACT- 
R  (Anderson  1993;  Anderson  &  Lebiere  1998;  Anderson  & 
Lebiere  2003);  CLARION  (Sun  2001);  Polyscheme  (Cas- 
simatis  2002;  Cassimatis  et  al.  2004)).  What  makes  the 
RASCALS  cognitive  architecture  distinctive?  There  is  in¬ 
sufficient  space  here  to  convey  any  technical  detail  (for  more 
details,  see  (Bringsjord  forthcoming));  we  make  just  three 
quick  points  about  RASCALS,  to  wit: 

•  All  other  cognitive  architectures  we  know  of  fall  far  short 
of  the  expressive  power  of  RASCALS.  For  example, 
SOAR  and  ACT-R  struggle  to  represent  (let  alone  reason 
quickly  over)  textbook  problems  in  logic  (e.g.,  the  Wise 
Man  Problem  =  WMP)  but  in  RASCALS  such  representa¬ 
tions  are  effortless  (see  (Arkoudas  &  Bringsjord  2005)  for 
the  solution  to  WMP  in  Athena,  included  in  RASCALS). 

•  The  great  challenge  driving  the  field  of  computational 
cognitive  modeling  (CCM)  is  to  unify  all  of  human  cogni¬ 
tion;  this  challenge  can  be  traced  back  to  the  birth  of  CCM 
in  the  work  of  Newell  1973.  Such  unification  is  achieved 
in  one  fell  swoop  by  RASCALS,  because  all  of  cognition 


'Coincidentally,  we  have  recently  learned  that  the  game  Shat¬ 
tered  World  for  the  X  Box  is  related  to  our  scenario. 


'Rensselaer  Advanced  Synthetic  Character  Architecture  for 
Logical  Systems 
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can  be  formalized  and  mechanized  in  logic  (though  do¬ 
ing  so  requires  some  very  complicated  logics  well  beyond 
first-order  logic,  as  in  (Arkoudas  &  Bringsjord  2005)). 

•  While  logic  has  been  criticized  as  too  slow  for  real-time 
perception-and-action-heavy  computation,  as  you  might 
see  in  first-person  shooter  (as  opposed  to  a  strategy  game, 
which  for  obvious  reasons  fits  nicely  with  the  paradigm  of 
logic-based  AI),  it  has  been  shown  that  RASCALS  is  so 
fast  that  it  can  enable  the  real-time  behavior  of  a  mobile 
robot.  We  have  shown  this  by  having  a  logic-based  mobile 
robot  successfully  navigate  the  wumpus  world  game,  a 
staple  in  AI.  (See  Figures  4  and  5.) 


Figure  4:  The  Wumpus  World  Game 


Solid  Performance 
Based  on  Logic 


2x2  3  x  3  4x4  5x5  6x  6  7x7  8x8  9x9  10x10 


Figure  5:  Performance  of  a  RASCALS-Powered  Robot  in 
the  Wumpus  World 

To  show  part  of  the  underlying  structure  of  E  in  con¬ 
nection  with  the  attribute  Extreme  Expressivity,  we  now 
present  an  informal  version  of  the  formal  account  of  evil 


that  is  implemented  in  our  RASCALS  architecture.  This  ac¬ 
count  specifically  requires  logics  expressive  enough  to  han¬ 
dle  knowledge,  belief,  and  ethical  concepts.  These  logics  go 
well  beyond  first-order  logic;  details  and  an  implementation 
can  be  found  in  (Arkoudas  &  Bringsjord  2005).  In  the  sec¬ 
tion  “E:  The  Presentation  Level”  we  explain  the  technology 
that  allows  E  to  speak  naturally  in  English;  that  is,  we  show 
there  part  of  the  underlying  structure  of  E  associated  with 
Uses  Natural  Language. 

On  our  Formal  Account  of  Evil 

If  we  charitably  push  things  in  the  direction  of  formally  rep¬ 
resenting  a  definition  of  evil,7 8  then  we  can  understand  Fein- 
berg  2003  as  advancing  pretty  much  this  definition: 

Def  1  Person  s  is  evil  iff  there  exists  some  action  as  such  that 

1 .  performing  a  is  morally  wrong; 

2.  s  is  morally  blameworthy  for  performing  a; 

3.  s’ s  performing  a  causes  considerable  harm  to  others;  and 

4.  the  reasons  or  motives  for  s’s  performing  a,  along  with 
“the  elements  that  ground  her  moral  blameworthiness,” 
are  unintelligible. 

This  is  a  decent  starting  place,  but  for  us  there  are  prob¬ 
lems.  For  example,  imagine  that  E  invariably  fails  to  cause 
actual  harm.  Surely  he  would  still  qualify  as  evil  even 
if  he  were  a  bumbling  villain.  (If  the  knife  slipped  when 
he  attempted  decapitation,  he  would  still  be  just  as  black¬ 
hearted.)  This  means  that  clause  3  should  at  least  be  replaced 
by 

3/.  s  performs  a  in  the  hopes  of  causing  considerable  harm 
to  others 

But  even  this  new  definition,  for  reasons  we  don’t  have 
space  to  explain,  is  wholly  inadequate.  To  give  just  a  flavor 
for  what  E  is  currently  based  upon,  we  present  simply  our 
current  best  replacement  for  clause  4: 

4"  were  s  a  willing  and  open  participant  in  the  analysis  of 
reasons  and  motives  for  s’s  seeking  to  perform  a ,  it  would 
be  revealed  that  either 

(i)  these  reasons  and  motives  are  unintelligible,  or 

(ii)  s  seeks  to  perform  a  in  the  service  of  goal  g,  and 

(a)  the  anticipatable  side-effects  e  of  performing  a  are 
bad,  but  s  cannot  grasp  this,  or 

(b)  g  itself  is  appraised  as  good  by  s  when  it  is  in  fact 
bad. 

Just  this  clause  alone  required  much  sustained  analysis.  (For 
a  full  chronicle  of  the  evolution  of  a  formally  refined  defini¬ 
tion  of  betrayal  from  a  rough  starting  one,  see  the  chapter 
“Betrayal”  in  (Bringsjord  &  Ferrucci  2000).) 

Keep  in  mind  that  this  is  still  informal,  kept  that  way  in  the 
interests  of  easing  exposition.  In  the  RASCALS-based  im¬ 
plementation  of  E,  evil  must  be  expressed  in  purely  formal 
form,  which  requires,  again,  that  we  use  advanced  logics  of 
belief,  knowledge,  and  obligation.9 

7Feinberg’s  work  is  informal,  and  not  suitable  for  direct  use  in 
AI  and  computer  science. 

8  Or  omission. 

9For  a  look  at  the  deontic  logic  (i.e.,  the  logic  of  ethical  con¬ 
cepts)  we  are  relying  upon,  see  (Horty  2001).  Our  mechanization 
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Keep  in  mind  as  well  that  we’re  not  claiming  that  we  have 
the  perfect  definition  of  evil.  Some  may  object  to  our  defi¬ 
nition.  and  some  of  their  objections  may  be  trenchant.  But 
the  important  point  is  to  see  how  rich  evil  is  —  to  see  that  it 
involves  all  kinds  of  highly  cognitive  powers  and  concepts 
that  simply  aren’t  found  in  today’s  synthetic  characters.  To 
be  evil,  one  has  to  have  beliefs,  desires,  and  one  has  to  have 
a  lot  of  knowledge.  The  detailed  configuration  of  these  el¬ 
ements  may  not  be  exactly  as  we  claim  they  ought  to  be, 
but  no  one  can  deny  that  the  elements  are  needed.  Without 
them,  a  synthetic  character  who  is  supposed  to  be  evil  is  only 
a  fake  shell.  And  in  the  end.  the  shell  will  be  revealed  to  be 
a  shell:  the  illusion,  at  some  point,  will  break  down. 

How  Does  E  Talk? 

As  everyone  knows,  once  the  daunting  challenge  of  render¬ 
ing  consciousness  in  computational  terms  is  put  aside,  the 
greatest  remaining  challenge  is  that  of  giving  an  advanced 
synthetic  character  the  power  to  communicate  in  a  natural 
language  (English,  French,  etc.)  at  the  level  of  a  human  per¬ 
son.  As  you'll  recall,  communicative  capacity  is  one  of  the 
clauses  in  the  definition  of  personhood  presented  above.  A 
plausible  synthetic  character  must  necessarily  communicate 
in  a  fluid,  robust  manner.  How,  then,  is  such  a  rich  form  of 
communication  implemented  in  E? 

Reconciling  Knowledge  Representation  and  NLG 

E  speaks  by  parsing  and  processing  formal  knowledge;  he 
develops  an  ontology  based  on  internal  and  external  queries, 
and  then  reasons  over  his  knowledge  to  produce  meaningful 
content.  This  content  is  then  sent  to  his  NLG  module,  trans¬ 
lated  into  English,  and  finally  presented  to  the  user.  Before 
we  examine  what  goes  on  inside  E's  NLG  module,  let’s  take 
a  moment  to  examine  how  E  produces  “meaningful  content.” 

When  we  ask  E  a  question,  we  are  clearly  interested  in 
an  answer  that  is  both  relevant  and  meaningful,  an  answer 
indistinguishable  from  those  given  by  a  real  person.  Assum¬ 
ing  we  have  incomplete  knowledge,  suppose  we  ask  of  E, 
“Is  John  dangerous?”  E  approaches  this  question  through 
formal  logical  analysis.  The  idea  is  to  have  E  determine 
incontrovertibly  whether  John  is  dangerous  or  not.  So,  for 
instance,  suppose  E’s  knowledge  base  includes  the  follow¬ 
ing  three  facts: 

1.  Dangerous  people  have  automatic  weapons. 

2.  John  has  a  Beretta  AR-70  assault  rifle. 

3.  The  Beretta  AR-70  assault  rifle  is  an 
automatic  weapon. 

None  of  the  information  above  explicitly  tells  E  whether 
John  is  dangerous  or  not,  but  clearly,  when  presented 
the  above  query,  we  want  E  to  answer  with  an  emphatic 
“Yes.”  Still,  the  answer  itself  is  not  enough.  To  ensure 
that  E  understands  the  nature  of  the  question  as  well  as 
the  information  he  is  dealing  with,  he  must,  upon  request, 
provide  a  justification  for  every  answer.  The  justification 

of  this  this  logic  will  be  presented  at  the  AAAI  November  2005 
Fall  Symposium  on  Machine  Ethics.  The  paper  is  available  online 
at  http://kryten.mm.rpi.edu/FS605ArkoudasAndBringsjord.pdf. 


presented  to  the  user  is  a  formal  proof,  translated  into 
English.  Thus,  E  could  answer  as  follows: 

John  is  in  fact  dangerous  because  he  has 
a  Beretta  AR-70  assault  rifle.  Since  a 
Beretta  AR-70  assault  rifle  is  an  automatic 

WEAPON,  AND  SINCE  DANGEROUS  PEOPLE  HAVE 

automatic  weapons,  it  follows  that  John  is 
DANGEROUS . 

Content  is  thus  generated  in  the  form  of  a  formal  proof.  In 
general,  the  proofs  generated  will  be  more  complex  (they 
will  use  larger  knowledge  bases)  and  more  sophisticated 
(they  will  use  deontic  and  epistemic  logic). 

While  the  example  is  simple  and  rudimentary  (that  is,  it 
makes  use  of  only  first-order  logic  and  a  small  knowledge 
base),  it  demonstrates  that  E  is  taking  heed  of  his  knowledge 
to  generate  a  meaningful  reply.  In  the  RASCALS  architec¬ 
ture,  answering  “Yes”  to  the  query  above  implies  that  E  must 
in  fact  have  the  corresponding  knowledge,  an  implication 
that  does  not  hold  for  other  architectures. 

For  a  more  formal  method  of  analysis,  we  introduce 
the  “Knowledge  Code  Test”:  If  synthetic  character  C  says 
something  X  or  does  something  A'  designed  to  evoke  in 
the  mind  of  the  human  gamer/user  the  belief  that  C  knows 
F\ ,  P-2, . . then  we  should  find  a  list  of  formulas,  or  the 
equivalent,  corresponding  to  Pi,  P-2,  ■  ■  .  in  the  code  itself. 
The  characters  in  Figure  1  would  fail  such  a  test,  as  would 
characters  built  on  the  basis  of  Champandard’s  specifica¬ 
tions.  An  FSA.  as  a  matter  of  mathematical  fact,  has  no 
storage  capability.  A  system  with  power  that  matches  that  of 
a  full  Turing  machine  is  needed  to  pass  the  Knowledge  Code 
Test  (Lewis  &  Papadimitriou  1981). 

But  formal  proofs  are  oftentimes  too  detailed  to  be  of  in¬ 
terest.  Before  we  can  even  begin  translating  a  proof  into 
an  English  justification,  we  need  verify  that  its  level  of  ab¬ 
straction  is  high  enough  that  it  is  easy  to  read  and  under¬ 
stand.  After  all,  formal  natural  deduction  proofs  are  difficult 
and  tedious  to  read.  To  represent  proofs  at  a  more  wholis- 
tic,  abstract  level,  we  utilize  the  denotational  proof  language 
known  as  Athena  (Arkoudas  2000).  Athena  is  a  program¬ 
ming  language,  development  environment,  and  interactive 
proof  system  that  evaluates  and  processes  proofs  as  input. 
Its  most  prominent  feature  is  its  ability  to  present  proofs  in 
an  abstract,  top-level  manner,  isomorphic  to  that  of  a  natu¬ 
ral  argument  a  human  might  use.  By  developing  proofs  in 
Athena  at  this  level,  a  level  high  enough  to  be  of  interest  to 
a  human  reader,  we  can  be  sure  that  the  language  generated 
from  our  NLG  module  is  at  precisely  the  level  of  abstraction 
we  desire  —  neither  too  detailed  nor  too  amorphous. 

It’s  now  time  to  look  at  precisely  how  English  is  generated 
from  a  formal  proof. 

Proof-based  Natural  Language  Generation 

Very  few  researchers  are  experimenting  with  the  rigorous 
translation  of  formal  proofs  into  natural  language10.  This  is 

lnAn  example  of  one  such  team  is  a  research  group  at  the  Uni¬ 
versity  of  Saarlande.  The  group  had,  until  1997.  been  developing 
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particularly  odd  when  one  considers  the  benefits  of  such  a 
program.  Natural  deduction  proofs,  provided  that  they  are 
developed  in  a  sensible  manner,  are  already  poised  for  ef¬ 
ficient  translation.  They  require  absolutely  no  further  doc¬ 
ument  structuring  or  content  determination.  That  is,  docu¬ 
ment  planning,  as  defined  by  (Reiter  &  Dale  2000),  is  com¬ 
pletely  taken  care  of  by  using  formal  proofs  in  the  first  place. 

Our  NLG  module  receives  as  input  a  formal  proof  and  re¬ 
turns  as  output  English  text.  The  English  generated  is  an 
isomorph  of  the  proof  received.  The  structure  of  the  justi¬ 
fication.  then,  is  precisely  the  same  as  the  structure  of  the 
proof.  If  the  justification  uses  reductio  ad  absurdum  in  the 
middle  of  the  exposition,  then  you  can  be  sure  that  there's  a 
proof  by  contradiction  in  the  middle  of  the  formal  proof. 

Formal  proofs  are  constructed  from  various  different  sub¬ 
proofs.  A  proof  by  contradiction  is  one  such  example  of 
a  type  of  subproof,  but  there  are  of  course  many  others. 
Our  system  breaks  a  proof  down  to  its  constituent  subproofs, 
translating  each  subproof  from  the  top  down.  For  example, 
assume  the  following: 

1.  Chicago  is  a  target  or  New  York  is  a 

TARGET 

2.  If  Chicago  is  a  target,  millions  will  die. 

3.  If  New  York  is  a  target,  millions  will 

die  . 

To  deduce  something  meaningful  from  this  information, 
we'll  use  a  proof  by  cases.  Our  system  translates  this  proof 
form  as  follows: 

Recall  that  Chicago  or  New  York  is  a  target. 
Each  case  produces  the  same  conclusion;  that 
is,  if  Chicago  is  a  target  then  millions 
will  die,  and  if  New  York  is  a  target  then 
millions  will  die.  It  follows  that  millions 

WILL  DIE . 

Predictably,  documents  produced  in  this  manner,  even 
when  presented  at  a  level  abstract  enough  to  make  sense 
to  a  layperson,  are  rigid  and,  well,  inhuman.  They  use  the 
same  phrases  over  and  over  again,  they  lack  fluidity,  and 
they  are  completely  divorced  of  grace  and  wit.  To  boot, 
they  disregard  contextual  information.  Merely  translating 
constituent  subproofs  to  English  will  not  produce  natural 
English. 

Nevertheless,  this  methodology  provides  a  foundation  for 
more  sophisticated  development.  Once  constituent  sub¬ 
proofs  are  translated  properly,  they  are  sent  to  a  microplan¬ 
ning  system  that  maps  particular  subproofs  to  discourse  rela¬ 
tions  (Elovy  1993).  This  mapping  is  known  as  a  message  and 
is  not  isomorphic.  While  the  structure  of  the  overall  proof 
is  preserved  in  the  final  document,  individual  subproofs  are 
not  treated  with  the  same  stringency.  They  can  be  molded 
and  fitted  to  a  number  of  different  discourse  relations  for  the 
sake  of  fluidity.  Two  more  steps  remain  before  natural  lan¬ 
guage  can  be  produced. 

a  system  called  PROVERB  (Huang  &  Fiedler  1997).  Their  ap¬ 
proach  to  proof-based  translation  was  unique  and  extremely  influ¬ 
ential,  though  their  project  was  largely  unsuccessful. 


Lexicalization  is  the  process  by  which  a  lexicon  of 
words  is  selected  and  mapped  onto  its  symbolic  coun¬ 
terparts.  The  content  implicit  in  the  proof,  structured 
through  subproof  analysis  and  discourse  relations,  needs 
to  be  lexicalized  before  it  can  be  presented  as  English 
text.  That  is,  exact  words  and  phrases  must  be  chosen 
to  represent  relationships  and  predicates.  For  instance, 
Target  (Chicago)  must  be  translated  to  Chicago  is 
a  target  and  Beretta  (  John)  must  be  translated  to 
John  has  a  Beretta  before  we  can  move  on  to  glu¬ 
ing  everything  together.  The  only  way  this  can  happen  is 
if  a  lexical  database  such  as  WordNet  (Miller  1995)  is  aug¬ 
mented  with  domain-specific  lexicalizations  such  as  those 
specifying  how  to  lexicalize  “Beretta  AR-70.” 

For  even  more  fluidity,  it's  necessary  to  avoid  referring  to 
the  same  entities  with  the  same  phraseology.  At  the  very 
least,  pronouns  should  be  substituted  when  referring  to  re¬ 
peated  concepts,  persons,  places,  and  objects.  These  substi¬ 
tutions  are  known  as  referring  expressions,  and  need  to  be 
generated  to  truly  produce  fluid,  humanlike  English. 

Fortunately,  once  the  above  issues  are  resolved,  the  infor¬ 
mation  gathered  therein  can  be  plugged  easily  into  a  surface 
realizer  such  as  KPML  (Bateman  1997).  In  this  fashion, 
proof-based  NLG  allows  for  the  generation  of  both  struc¬ 
tured  and  expressive  expositions. 

E:  The  Presentation  Level 

To  concretize  our  representation  of  evil  (as  in  demos,  e.g.; 
see  the  final  section  of  the  paper),  we  show  E;  a  realistic  real¬ 
time  presentation  of  an  evil  talking  head  in  the  formal  sense. 
In  order  to  give  E  a  realistic  look,  a  range  of  facial  expres¬ 
sions,  and  a  flexible  response  to  input,  we  simulate  a  subset 
of  the  muscles  in  the  face.  Each  muscle  in  our  model  can 
contract,  perturbing  the  underlying  triangle  mesh.  Our  sim¬ 
ulation  is  based  largely  on  that  presented  in  (Waters  1987) 
and  we  have  taken  the  approach  of  implementing  the  model 
almost  entirely  in  a  vertex  shader.  A  parameterization  for 
the  tongue  similiar  to  (King  2001)  is  used.  A  module  for 
eye  movements  implements  ideas  presented  in  (Lee,  Badler, 
&  Badler  2002).  Finally,  we  simulate  subsurface  scatter¬ 
ing  on  the  skin  using  the  algorithm  of  (Sander,  Gosselin,  & 
Mitchell  2004).  Our  tool  is  shown  in  Figure  6. 

Our  Demos  @  GameOn! 

As  mentioned  above,  at  the  conference  we  will  allow  atten¬ 
dees  to  discuss  with  E  the  two  aforementioned  scenarios, 
and  this  interaction  will  show  our  approach  to  the  presen¬ 
tation  level  in  action,  and  will  manifest  our  formal  account 
of  evil  in  ordinary  conversation  that  is  based  on  our  NLG 
technology. 
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Abstract 

We  suggest  that  mechanized  multi-agent  deontic  logics  might 
be  appropriate  vehicles  for  engineering  trustworthy  robots. 
Mechanically  checked  proofs  in  such  logics  can  serve  to  es¬ 
tablish  the  permissibility  (or  obligatoriness)  of  agent  actions, 
and  such  proofs,  when  translated  into  English,  can  also  ex¬ 
plain  the  rationale  behind  those  actions.  We  use  the  logical 
framework  Athena  to  encode  a  natural  deduction  system  for  a 
deontic  logic  recently  proposed  by  Horty  for  reasoning  about 
what  agents  ought  to  do.  We  present  the  syntax  and  seman¬ 
tics  of  the  logic,  discuss  its  encoding  in  Athena,  and  illustrate 
with  an  example  of  a  mechanized  proof. 

Introduction 

As  machines  assume  an  increasingly  prominent  role  in  our 
lives,  there  is  little  doubt  that  they  will  eventually  be  called 
upon  to  make  important,  ethically  charged  decisions.  How 
can  we  trust  that  such  decisions  will  be  made  on  sound  ethi¬ 
cal  principles?  Some  have  claimed  that  such  trust  is  impos¬ 
sible  and  that,  inevitably,  Al  will  produce  robots  that  both 
have  tremendous  power  and  behave  immorally  (Joy  2000). 
These  predictions  certainly  have  some  traction,  particularly 
among  a  public  that  seems  bent  on  paying  good  money  to  see 
films  depicting  such  dark  futures.  But  our  outlook  is  a  good 
deal  more  optimistic.  We  see  no  reason  why  the  future,  at 
least  in  principle,  can’t  be  engineered  to  preclude  doomsday 
scenarios  of  malicious  robots  taking  over  the  world. 

One  approach  to  the  task  of  building  well-behaved  robots 
emphasizes  careful  ethical  reasoning  based  on  mechanized 
formal  logics  of  action,  obligation,  and  permissibility;  that  is 
the  approach  we  explore  in  this  paper.  It  is  a  line  of  research 
in  the  spirit  of  Leibniz’s  famous  dream  of  a  universal  moral 
calculus  (Leibniz  1984): 

When  controversies  arise,  there  will  be  no  more  need 
for  a  disputation  between  two  philosophers  than  there 
would  be  between  two  accountants  [computistas].  It 
would  be  enough  for  them  to  pick  up  their  pens  and  sit 
at  their  abacuses,  and  say  to  each  other  (perhaps  having 
summoned  a  mutual  friend):  ‘Let  us  calculate.’ 

*We  gratefully  acknowledge  that  this  research  was  in  part  sup¬ 
ported  by  Air  Force  Research  Labs  ( AFRL),  Rome. 
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In  the  future  we  envisage,  Leibniz’s  “calculation”  would  boil 
down  to  formal  proof  and/or  model  generation  in  rigorously 
defined,  machine-implemented  logics  of  action  and  obliga¬ 
tion. 

Such  logics  would  allow  for  proofs  establishing  that: 

1.  Robots  only  take  permissible  actions;  and 

2.  all  actions  that  are  obligatory  for  robots  are  actually  per¬ 
formed  by  them  (subject  to  ties  and  conflicts  among  avail¬ 
able  actions). 

Moreover,  such  proofs  would  be  highly  reliable  (i.e.,  have 
a  very  small  “trusted  base”),  and  explained  in  ordinary  En¬ 
glish. 

Clearly,  this  remains  largely  a  vision.  There  are  many 
thorny  issues,  not  least  among  which  are  criticisms  regard¬ 
ing  the  practical  relevance  of  such  formal  logics,  efficiency 
issues  in  their  mechanization,  etc.;  we  will  discuss  some  of 
these  points  shortly.  Nevertheless,  mechanized  ethical  rea¬ 
soning  remains  an  intriguing  vision  worth  investigating. 

Of  course  one  could  also  object  to  the  wisdom  of  logic- 
based  AI  in  general.  While  other  ways  of  pursuing  AI 
may  well  be  preferable  in  certain  contexts,  we  believe  that 
in  this  case  a  logic-based  approach  (Bringsjord  &  Fer- 
rucci  1998a;  1998b;  Genesereth  &  Nilsson  1987;  Nilsson 
1991;  Bringsjord,  Arkoudas,  &  Schimanski  foithcoming)  is 
promising  because  one  of  the  central  issues  here  is  that  of 
trust — and  mechanized  formal  proofs  are  perhaps  the  single 
most  effective  tool  at  our  disposal  for  establishing  trust. 

Deontic  logic,  agency,  and  action 

In  standard  deontic  logic  (Chellas  1980;  Hilpinen  2001; 
Aqvist  1984),  or  just  SDL,  the  formula  Q)P  can  be  inter¬ 
preted  as  saying  that  it  ought  to  be  the  case  that  P,  where 
P  denotes  some  state  of  affairs  or  proposition.  Notice  that 
there  is  no  agent  in  the  picture,  nor  are  there  actions  that  an 
agent  might  perform.  This  is  a  direct  consequence  of  the 
fact  that  SDL  is  derived  directly  from  standard  modal  logic, 
which  applies  the  possibility  and  necessity  operators  O  and 
□  to  formulae  standing  for  propositions  or  states  of  affairs. 
For  example,  the  deontic  logic  D*  has  one  rule  of  inference, 
viz., 

-P  — >  Q 

OP  -*■  OQ 
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and  three  axiom  schemas: 

•  (OP  a  OQ)  -*•  0(P  a  Q) 

•  O  T  («  “That  which  must  be  is  obligatory.”) 

•  -i  O  -L  (~  “Nothing  impossible  is  obligatory.”) 

While  D*  has  some  desirable  properties,  it  and  its  rela¬ 
tives  are  plagued  by  various  paradoxes  (Hilpinen  2001),  and, 
more  importantly  given  present  purposes,  these  logics  aren’t 
targeted  at  formalizing  the  concept  of  actions  being  oblig¬ 
atory  (or  permissible  or  forbidden)  for  an  agent.  Interest¬ 
ingly,  deontic  logics  that  have  agents  and  their  actions  in 
mind  do  go  back  to  the  very  dawn  of  this  subfield  of  logic 
(von  Wright  1951),  but  only  recently  has  an  “Al-friendly” 
semantics  been  proposed  (Belnap,  Perloff,  &  Xu  2001; 
Horty  2001)  and  corresponding  axiomatizations  been  inves¬ 
tigated  (Murakami  2004). 

We  have  used  the  Athena  logical  framework  (briefly  dis¬ 
cussed  in  the  next  section)  to  encode  a  natural  deduction 
calculus  for  a  modern  logic  of  agent  action  and  obligation 
developed  by  Horty  and  axiomatized  by  Murakami  in  order 
to  investigate  mechanical  deontic  reasoning. 

The  ideal  conditions  for  building  “ethical  robots”  via  a 
logic-based  approach  to  A1  would  be  as  follows:  we  would 
have  an  expressive  deontic  logic  C  of  high  practical  rele¬ 
vance,  and  an  efficient  algorithm  for  determining  theorem- 
hood  in  C.  That  algorithm  could  then  be  built  into  the 
robot  (perhaps  implemented  directly  on  its  hardware),  and 
the  robot  would  only  take  an  ethically  charged  action  if  it 
could  formally  prove  that  the  action  is  permissible.  Unfortu¬ 
nately,  there  is  a  legendarily  strong  tension  between  expres¬ 
siveness  and  efficiency,  and  so  it  is  certain  that  these  ideal 
conditions  will  never  obtain.  For  expressiveness,  we  will 
likely  need  highly  hybrid  modal  and  deontic  logics  that  are 
at  least  first-order,  which  means  that  theoremhood  in  such 
logics  will  be  undecidable.  Even  for  decidable  logics,  such 
as  the  zero-order  version  of  Horty’s  system  1  that  we  present 
in  this  paper,  decision  procedures  are  likely  to  be  of  inordi¬ 
nate  computational  complexity. 

Therefore,  we  must  reconcile  ourselves  to  the  possibil¬ 
ity  that  a  robot  might  not  be  able  by  itself  to  pass  judgment 
on  certain  actions  that  it  is  contemplating;  and  that  instead 
of  a  single  monolithic  decision  procedure  for  deontic  the¬ 
oremhood  (or  validity,  assuming  the  logic  is  complete),  the 
robot  might  instead  need  to  be  armed  with  a  knowledge  base 
of  lemmas  and  a  panoply  of  tactics ,  each  capable  of  set¬ 
tling  certain  restricted  subclasses  of  deontic  questions.  This 
might  well  turn  out  to  be  sufficient  in  practice.  If  and  when 
a  complicated  proposition  arises  that  ethically  paralyzes  the 
robot,  humans  could  intervene  to  settle  the  situation  as  they 
see  fit.  The  logical  framework  that  we  have  used  to  mech¬ 
anize  Horty’s  logic,  Athena  (Arkoudas  2003),  facilitates  the 
formulation  of  lemmas  and  of  highly  reliable  tactics. 

Athena 

Athena  (Arkoudas  2003)  is  a  new  interactive  theorem  prov¬ 
ing  system  for  polymorphic  multi-sorted  first-order  logic 

'Proved  complete  and  decidable  by  Murakami  (Murakami 
2004). 


that  incorporates  facilities  for  model  generation,  automated 
theorem  proving,  and  structured  proof  representation  and 
checking.  It  also  provides  a  higher-order  functional  pro¬ 
gramming  language,  and  a  proof  abstraction  mechanism  for 
expressing  arbitrarily  complicated  inference  methods  in  a 
way  that  guarantees  soundness,  akin  to  the  tactics  and  tac- 
ticals  of  LCF-style  systems  such  as  HOL  (Gordon  &  Mel- 
ham  1993)  and  Isabelle  (Paulson  1994).  Proof  automation  is 
achieved  in  two  ways:  first,  through  user-formulated  proof 
methods;  and  second,  through  the  seamless  integration  of 
state-of-the-art  ATPs  such  as  Vampire  (Voronkov  1995)  and 
Spass  (Weidenbach  2001)  as  primitive  black  boxes  for  gen¬ 
eral  reasoning.  For  model  generation,  Athena  integrates 
Paradox  (Claessen  &  Sorensson  2003),  a  new  highly  effi¬ 
cient  model  finder.  For  proof  representation  and  checking, 
Athena  uses  a  block-structured  Fitch-style  natural  deduc¬ 
tion  calculus  (Pelletier  1999)  with  novel  syntactic  constructs 
and  a  formal  semantics  based  on  the  abstraction  of  assump¬ 
tion  bases  (Arkoudas  2000).  Most  interestingly,  a  block- 
structured  natural  deduction  format  is  used  not  only  for  writ¬ 
ing  proofs,  but  also  for  writing  tactics  (methods).  This  is  a 
novel  feature  of  Athena.  Tactics  in  this  style  are  consider¬ 
ably  easier  to  write  and  remarkably  useful  in  making  proofs 
more  modular  and  abstract. 

Athena  has  been  used  to  implement  a  proof-emitting  op¬ 
timizing  compiler  (Rinard  &  Marinov  1999);  to  integrate 
model  checking  and  theorem  proving  for  relational  reason¬ 
ing  (Arkoudas  el  al.  2003);  to  implement  various  “certi¬ 
fying”  algorithms  (Arkoudas  &  Rinard  2004);  to  verify  the 
core  operations  of  a  Unix-like  file  system  (Arkoudas  et  al. 
2004);  to  prove  the  correctness  of  dataflow  analyses  (Hao 
2002);  and  to  reason  about  generic  software  (Musser  2004). 
A  concise  presentation  of  Athena’s  syntax  and  semantics  can 
be  found  elsewhere  (Arvizo  2002). 

Horty’s  logic  and  its  Athena  encoding 

Murakami  (Murakami  2004)  presents  an  axiomatization  of 
Horty’s  utilitarian  formulation  of  multi-agent  deontic  logic 
(Horty  2001),  and  shows  it  decidable  by  proving  that  it  has 
the  finite  model  property.  In  this  section  we  develop  an  alter¬ 
native,  sequent-based  natural-deduction  formulation  of  Mu¬ 
rakami’s  system.  The  logic  is  encoded  in  Athena,  which 
is  then  used  as  a  metalanguage  in  order  to  reason  about 
the  encoded  object  language;  we  have  used  this  methodol¬ 
ogy  successfully  with  other  intensional  logics  (Arkoudas  & 
Bringsjord  2005).  In  what  follows  we  briefly  review  the  ab¬ 
stract  syntax  and  semantics  of  the  logic,  and  then  present  our 
formulation  of  a  natural  deduction  system  for  it. 

We  use  the  letters  P,  Q,  R, . . .,  to  designate  arbitrary 
propositions,  built  according  to  the  following  abstract  gram¬ 
mar: 

P  ::=  A|Tj_L|^P|PAQ|PVQ|P=M3 
|  DP  |  OP  |  [a  cstit:  P]  |  ©  [a  cstit:  P ] 

where  A  and  a  range  over  a  countable  set  of  atomic 
propositions  (“atoms”)  and  a  primitive  domain  of  agents, 
respectively.  Propositions  of  the  form  [a  cstit:  P]  and 
©  [a  cstit:  P]  are  read  as  “a  sees  to  it  that  P”  and  “a 


31 


FhP  rhQ  [A-/] 

r  h  p  a  q 


r  h  P  A  Q  [A-Bi] 

r  hp 


r  h  p  A  Q  [a -e2] 
r  h  q 


rhP  [V-/!] 

r  h  p  v  q 


r  h  Q _  [V-J2] 

r  h  p  v  Q 


r  h  v  p2  r,  Pi  h  q  r,  p2  n  q  [v.£] 
r  h  q 


r'pl~Q  [=>-/] 

r  h  p=^q 


rt-p=M5  ri-p 
r  h  q 

-£] 

[-L -I] 


[  =>-E] 


r  i -  p 

ri-  P  A  —>p 
rhi 


r,p  h  p 


rhnp 

r  h  t 

r  h  p 
rur'hP 


[T -/] 


[Di/wrion] 


Figure  1:  Inference  rules  for  the  propositional  connectives. 


ought  to  see  to  it  that  P,”  respectively.  2  We  stress  that 
©  [a  cstit:  P }  is  not  read  as  “It  ought  to  be  the  case  that  a 
sees  to  it  that  P.”  That  is  the  classic  Meinong-Chisholm 
“ought-to-be”  analysis  of  agency,  captured  by  another  for¬ 
mula  altogether,  O  [ a  cstit:  P],  where  O  is  the  non-agent- 
oriented  “ought”  operator  similar  to  what  is  found  in  SDL. 
In  Horty’s  semantics,  O  [a  cstit:  P]  and  ©  [a  cstit:  P)  are 
not  equivalent  statements;  neither  implies  the  other  (Horty 
2001).  In  general,  the  operator  Q,  taken  over  from  SDL,  ap¬ 
plies  to  P  just  in  case  P  holds  in  each  of  the  best  worlds.  As 
Horty  explains,  an  analogue  to  this  basic  idea  is  expressed 
by  ©  [a  cstit:  P],  because  this  locution  holds  whenever  P  is 
ensured  by  each  of  the  agent’s  best  actions.  (We  have  lit¬ 
tle  use  for  the  standard  obligation  operator  O  and  hence  we 
omit  it  from  our  formulation,  although  it  could  be  easily  in¬ 
cluded.) 

The  formal  semantics  are  given  on  the  basis  of  the  the¬ 
ory  of  indeterminate  branching  time  (Prior  1967;  Thomason 
1984),  augmented  with  constructs  for  dealing  with  agent  ac¬ 
tions.  The  usual  Kripke  frames  of  modal  logic  are  replaced 
by  deontic  stit  frames.  A  deontic  stit  frame  has  the  following 
components: 

•  A  set  of  moments  M,  along  with  a  strict  partial  order  <  on 


2The  ‘c’  in  cstit  stands  for  “Chellas.”  Horty  (Horty  2001)  at¬ 
tributes  the  naming  to  the  fact  that  cstit  is  analogous — though  not 
identical — to  an  operator  introduced  by  Brian  Chellas  in  his  1969 
doctoral  dissertation  (Chellas  1969).  There  are  other  stit  operators 
in  the  literature,  e.g.,  the  achievement  stit  (“astit”),  the  deliberative 
stit  (“dstit”),  etc. 


M  (i.e.,  <  is  irreflexive  and  transitive,  and  hence  asym¬ 
metric  as  well).  A  maximal  linearly  ordered  subset  of  M 
is  called  a  history.  The  set  of  all  histories  containing  a 
moment  to  €  M  is  written  as  Hm. 

•  A  set  A  of  agents. 

•  A  binary  function  Choice  that  maps  any  given  agent  a  and 
moment  to  into  a  partition  Choice(a,m)  of  Hm.  This 
function  must  satisfy  two  constraints:  independence  of 
agents ,  and  no  choice  between  undivided  histories',  see 
(Horty  2001)  for  details. 

•  For  each  m  £  M,  a  utility  function  Vm  from  Hm  into 
some  partially  ordered  set  of  values  (typically  the  real 
numbers). 

The  semantics  are  given  with  respect  to  moment/history 
pairs.  Specifically,  a  deontic  slit  model  is  a  deontic  stit  frame 
along  with  a  truth  valuation  that  maps  each  pair  (to,  h)  with 
to  €  M,  h  £  Hm  into  a  subset  of  atomic  propositions  (intu¬ 
itively,  these  are  the  atoms  that  are  true  at  the  index  (to,  h)). 
Given  a  deontic  stit  model  A4  and  a  moment/history  pair 
(to,  h)  (with  h  £  Hm),  we  write  M  |=  ^  P  to  mean 

that  A4  satisfies  proposition  P  at  index  (m,  h).  The  defini¬ 
tion  of  A4  \=^m  P  is  given  by  induction  on  the  struc¬ 
ture  of  P.  The  cases  of  atoms  and  propositional  combina¬ 
tions  are  standard.  Cstit  propositions  are  handled  as  follows: 
M  \={m,h)  I®  ™tit:  P)  iff 

M  ^  (to,  h')  P 

for  every  h'  £  block(m ,  a,  h),  where  block(m,  a,  h)  is  the 
unique  block  (equivalence  class)  containing  h  in  the  parti¬ 
tion  Choice(a,m )  of  We  refer  the  reader  to  (Horty 
2001)  for  the  semantics  of  ©  [a  cstit:  P], 

A  sequent  ThP  consists  of  a  context  1  (a  finite  set  of 
propositions)  and  a  proposition  P.  Intuitively,  this  states 
that  P  follows  from  T.  We  write  1.  P  (or  P,  1)  as  an  ab¬ 
breviation  for  r  U  {P}.  The  sequent  calculus  that  we  use 
consists  of  a  collection  of  inference  rules  for  deriving  judg¬ 
ments  of  the  form  T  h  P.  Figure  1  shows  the  inference  rules 
that  deal  with  the  standard  propositional  connectives.  These 
are  the  usual  introduction  and  elimination  rules  for  each  con¬ 
nective,  in  addition  to  reflexivity  and  dilution  (weakening). 
Further,  we  have  thirteen  rules  pertaining  to  the  modal  and 
deontic  operators,  shown  in  Figure  2.  [Pi],  [P4]  and  [P@] 
are  sequent  formulations  of  Kripke’s  “K”  axiom  for  the  op¬ 
erators  □  ,  cstit,  and  ©,  respectively.  [P2]  and  [P7]  are  the 
usual  “T”  axioms  of  modal  logic  for  □  and  cstit.  [P3]  is  the 
“axiom  5”  for  □.  [Pg]  and  [Pg]  express  that  necessary  truths 
are  ensured  (if  P  is  necessary  then  every  agent  sees  to  it)  and 
obligatory.  [P10]  asserts  that  obligations  are  possible.  [P12] 
is  a  necessitation  rule  ensuring  that  all  tautologies  (propo¬ 
sitions  derivable  from  the  empty  context)  are  necessary.  (A 
similar  necessitation  rule  for  cstit  can  be  derived  from  [P8] 
in  tandem  with  [P12],  so  we  do  not  need  to  take  it  as  prim¬ 
itive.)  [P13]  says  that  if  a  seeing  to  P  strictly  implies  a 
seeing  to  Q,  then  if  a  ought  to  stit  P  then  a  also  ought  to 
stit  Q.  Finally,  [P5]  is  a  slightly  disguised  formulation  of 
the  standard  “axiom  5”  for  cstit.  It  is  provably  equivalent  to 
-i [a  cstit:  -P]  =>  [a  cstit:  ->[a  cstit:  ~^P}} 
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-  [Ri] 

r  h  0(P  =>Q)=>  (OP  =>  DQ) 


-  [R2]  -  [R3] 

ri-np^p  n-op^nop 


-  [R4] 

r  h  [a  cstit:  P  =4>  Q]  =>■  ([a  cstit:  P]  =>  [a.  cstit:  Q] ) 


-  [#5] 

r  I - 1  [a  cstit:  P]  =>■  [a  cstit:  — «  [qj  cstit:  P]] 

-  [Re] 

rh  0[a  cstit:  P  =>•  Q]  =>• 

(O  [a  cstit:  P]  =>  ©  [a  cstit:  Q] ) 

-  [R7] 

r  h  [a  cstit:  P]  =>  P 


-  [Re] 

r  h  DP  =»  [a  cstit:  P] 


-  IR9] 

r  h  DP  =►©  [a  cstit:  P] 


-  [ Rio ] 

r  h  ©  [a  cstit:  P]  =4*  O  [ex.  cstit:  P] 


-  [-R11] 

T  h  (□  0  [a  cstit:  P])  V  (□  -1©  [a  cstit:  P]) 


r  h  op 

-  [^13] 

r  h  □  ([a  cstit:  P]  =>  [ck  cstit:  Q])  =>- 
(©  [a  cstit:  P]  =4*  ©  [a  cstit:  Q]) 


Figure  2:  Inference  rules  for  the  deontic  operators. 


which  is  of  the  exact  same  form  as  the  “axiom  5”: 


OP  =>  □  OP 

once  we  realize  that  OP  stands  for  ->□  ->P. 

Our  Athena  formalization  introduces  a  domain  of  agents 
and  a  datatype  that  captures  the  abstract  syntax  of  the  propo¬ 
sitions  of  the  logic: 

(datatype  Prop 
False 
True 

(Atom  Boolean) 

(If  Prop  Prop) 

(Not  Prop) 

(And  Prop  Prop) 

(Or  Prop  Prop) 

(Stit  Agent  Prop) 

(OughtToStit  Agent  Prop) 

(Nec  Prop) 

(Pos  Prop) ) 

We  proceed  to  introduce  a  binary  relation  sequent  that 


may  obtain  between  a  finite  set  of  propositions  and  a  sin¬ 
gle  proposition: 

(declare  sequent  (->  (  (FSet-Of  Prop)  Prop) 

Boolean) ) 

Here  FSet-Of  is  a  unary  sort  constructor:  for  any  sort 
T,  (FSet-Of  T)  is  a  new  sort  representing  the  set  of 
all  finite  sets  of  elements  of  T.  Finite  sets  are  built  with 
two  polymorphic  constructors:  the  constant  null,  repre¬ 
senting  the  empty  set;  and  the  binary  constructor  insert, 
which  takes  an  element  x  of  sort  T  and  a  finite  set  S  (of 
sort  (FSet-Of  T) )  and  returns  the  set  {x}  U  S.  We  also 
have  all  the  usual  set-theoretic  operations  available  (union, 
intersection,  etc.). 

The  intended  interpretation  is  that  if  (sequent  S  P) 
holds  for  a  set  of  propositions  S  and  a  proposition  P,  then 
the  sequent  S'  h  P  is  derivable  in  the  logic  via  the  above 
rules.  Accordingly,  we  introduce  axioms  capturing  those 
rules.  For  instance,  the  conjunction  introduction  rule  and 
rule  [Pio]  are  represented,  respectively,  by  the  following 
two  axioms: 

(define  And-I 

(forall  ?Gamma  ?P  ?Q 

(if  (and  (sequent  ?Gamma  ?P) 

(sequent  ?Gamma  ?Q) ) 

(sequent  ?Gamma  (And  ?P  ?Q) ) ) ) ) 

(define  RIO 

(forall  ?Gamma  ?a  ?P 
(sequent  ?Gamma 

(If  (OughtToStit  ?a  ?P) 

(Pos  (Stit  ?a  ?P) ) ) ) ) 

Note  the  object/meta-level  distinction  between,  e.g.,  and 
and  And.  The  former  is  a  native  Athena  propositional  con¬ 
structor,  i.e.,  part  of  the  metalogic,  whereas  the  latter  is  a 
propositional  constructor  of  the  encoded  object  logic. 

As  we  have  argued  elsewhere  (Arkoudas  &  Bringsjord 
2005),  such  a  direct  proof-theoretic  encoding  of  a  modal 
logic  in  a  first-order  logical  framework  such  as  Athena  car¬ 
ries  several  advantages: 

•  The  proofs  are  in  natural  deduction  format  and  hence  eas¬ 
ier  to  read,  write,  and  translate  into  English. 

•  Theorem  proving  is  facilitated  because  we  are  able 
to  leverage  state-of-the-art  automated  theorem  provers 
(ATPs)  such  as  Vampire  (Voronkov  1995)  and  Spass 
(Weidenbach  2001)  that  are  integrated  with  Athena.  Tac¬ 
tics  can  be  programmed  at  a  fairly  high  level  of  abstrac¬ 
tion,  with  tedious  details  outsourced  to  the  ATPs. 

•  Because  we  have  explicitly  encoded  the  abstract  syntax 
of  the  logic,  we  are  able  to  quantify  over  agents,  propo¬ 
sitions,  and  sequents.  This  provides  us  with  the  general¬ 
ization  benefits  of  higher-order  logic,  even  though  we  are 
working  in  a  first-order  system. 

Example 

As  a  simple  but  non-trivial  example,  we  present  the  Athena 
proof  of  the  following  “iterated  cstit”  result: 

[i a  cstit:  P ]  =>  [a  cstit:  [a  cstit:  P]]  (1) 


33 


pick-any  P  a 
begin 

SI  :=  (sequent 
null 

(If  (Not  (Stit  a  (Not  (Not  P) ) ) ) 
(Stit 
a 

(Not  (Stit 

a  (Not  (Not  P) ) ) ) ) ) ) 

from  R5; 

52  :=  (sequent 

null 

(If  (Not  (Stit 
a 

(Not  (Stit 
a 

(Not  (Not  P) ) ) ) ) ) 
(Not  (Not  (Stit 
a 

(Not  (Not  P) ) ) ) ) ) ) 
from  SI,  contrapositive; 

53  :=  prove  (sequent 

null 
(If  (Not 

(Not  (Stit 
a 

(Not  (Not  P) ) ) ) ) 
(Stit  a  (Not  (Not  P) ) ) ) ) ; 

54  :=  (sequent 

null 

(If  (Not  (Stit 
a 

(Not  (Stit 
a 

(Not  (Not  P) ) ) ) ) ) 
(Stit  a  (Not  (Not  P) ) ) ) ) 
from  S2,  S3,  transitivity; 

55  :=  prove  (sequent 

null  (Iff  P  (Not  (Not  P) ) ) ) ; 

56  :=  (sequent  null 

(Iff 

(Not  (Stit  a  (Not  (Stit  a  P) ) ) ) 
(Not  (Stit 
a 

(Not  (Stit 
a 

(Not  (Not  P) )))))) ) 
from  S5,  lemma-1.7; 


Figure  3:  Athena  proof  of  Lemma  1.8,  part  1. 


The  proof  is  easily  turned  into  a  tactic  that  can  be  applied  to 
any  given  agent  and  proposition. 

A  number  of  lemmas  are  used  in  the  proof.  Most  of  them 
express  straightforward  propositional  logic  tautologies  and 
are  proved  automatically  by  outsourcing  them  to  the  ATPs 
that  are  integrated  with  Athena.  For  instance,  the  first  four 
lemmas  below  respectively  express  the  transitivity  of  logical 
implication,  the  contrapositive  law,  the  cut,  and  disjunctive 
syllogism. 

Lemma  1.1  If  T  \-  P  ^  Q  and  r  h  Q  =>  R  then 


57  :=  (sequent 

null 

(If  (Not  (Stit  a  (Not  (Stit  a  P) ) ) ) 
(Not  (Stit 
a 

(Not  (Stit 
a 

(Not  (Not  P) )))))) ) 

from  S6,  Iff-Elim-1; 

58  :=  prove 

(sequent  null  (If  (Not  (Not  P) )  P) )  ; 

59  :=  (sequent 

null 

(If  (Stit  a  (Not  (Not  P) ) ) 

(Stit  a  P) ) ) 
from  S8,  lemma-1.6; 

(sequent  null 

(If  (Not  (Stit  a  (Not  (Stit  a  P) ) ) ) 
(Stit  a  P) ) ) 

from  S4,  S7,  S9,  lemma-1.5 
end 


Figure  4:  Athena  proof  of  Lemma  1.8,  part  2. 

r  h  P  =>  R. 

Lemma  1.2  IfY  1 1  ~>  (  J  then  !  I — Q  1 

Lemma  1.3  IfY\  h  P  and  T2 ,  P  h  Q  then  Ti  U  T2  h  Q. 

Lemma  1.4  r  h  (P1  V  P2)  =>  (^P2  =>  PI). 

Lemma  1.5  IfT  h  P'  =>  Q',  T  h  P  =>•  P',  andT  h  Q'  =>Q 
then  r  h  P  =>  Q. 

A  few  properly  deontic  lemmas  are  also  necessary: 

Lemma  1.6  For  all  agents  a  and  propositions  P  and  Q,  if 
0  h  P  =>  Q  then  0  h  [a  cstit:  P]  =>  [a  cstit:  Q\. 

Lemma  1.7  For  all  agents  a  and  propositions  P  and  Q,  if 
0  h  P  Q  then  01 — <[a  cstit:  P]  •<=>  -<[a  cstit:  Q\. 

Lemma  1.8  0  I — <[a  cstit:  [a  cstit:  P]]  =>■  [a  cstit:  P]  for 
all  a  and  P. 

Lemma  1.9  01 -  P  =>  ->[a  cstit:  — iP], 

Lemma  1.6,  Lemma  1.7,  and  Lemma  1.9  are  proved  auto¬ 
matically.  Lemma  1 .8  is  more  challenging  and  requires  user 
guidance.  Its  proof,  in  Athena’s  natural  deduction  system,  is 
shown  in  two  parts  in  Figure  3  and  in  Figure  4.  Very  brief 
explanations  of  the  pertinent  Athena  constructs  are  given  be¬ 
low  to  help  the  reader  follow  the  code.  For  a  more  thorough 
treatment  we  refer  the  reader  to  the  Athena  Web  site. 

An  Athena  deduction  D  is  always  evaluated  in  a  given  as¬ 
sumption  base — a  finite  set  of  propositions  that  are  assumed 
to  hold  for  the  purposes  of  D.  An  assumption  base  thus  rep¬ 
resents  our  “axiom  set”  or  “knowledge  base.”  Athena  starts 
out  with  the  empty  assumption  base,  which  then  gets  incre¬ 
mentally  augmented  with  the  conclusions  of  the  deductions 
that  the  user  successfully  evaluates.  Propositions  can  also 
be  explicitly  added  into  the  global  assumption  base  with  the 
top-level  directive  assert. 
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Evaluating  a  deduction  D  in  an  assumption  base  /3  will 
either  produce  a  proposition  F  (the  “conclusion”  of  D  in 
/3),  or  else  it  will  generate  an  error  or  will  diverge.  If  D 
does  produce  a  conclusion  F,  Athena’s  semantics  guarantee 
(3  \=  F,  i.e.,  that  F  is  a  logical  consequence  of  f3.  There  are 
several  syntactic  forms  that  can  be  used  for  deductions;  they 
form  a  complete  proof  system  for  polymorphic  multi-sorted 
first-order  logic. 

The  form  pick-any  introduces  universal  generalizations: 
pick-any  h  ■  ■  ■  In  begin  D  end  (for  arbitrary  subde¬ 
duction  D)  binds  the  names  I\  ■  ■  ■  /,,  to  fresh  variables 
i>i, . . . ,  vn  and  evaluates  D.  If  and  when  D  yields  a  con¬ 
clusion  F,  the  result  returned  by  the  entire  pick-any  is 
Vui  ,...,vn.F. 

The  body  of  the  proof  is  a  semicolon-separated  sequence 
of  steps  of  the  form 

Ii  :  —  }  •  •  •  ;  In  :  —  Dn 

where  I3  is  a  name  (identifier)  and  l)t  an  arbitrary  subproof. 
The  sequence  is  evaluated  by  recursively  evaluating  each  l)t 
in  turn,  j  =  1,2... .,  obtaining  a  conclusion  Fj,  binding  the 
name  I}  to  F;j ,  inserting  Fj  in  the  assumption  base,  and  then 
proceeding  with  the  next  step,  Ij+\  :=  Dj + 1  •  The  conclu¬ 
sion  of  the  entire  sequence  is  the  conclusion  of  the  last  step, 
Dn.  Note  that  the  last  step  is  not  named. 

A  common  proof  step  is  of  the  form 
F  from  Fi,...,Fk.  This  instructs  Athena  to  try  to 
automatically  derive  the  conclusion  F  from  the  given 
premises  F\ .....  />).  (all  k  of  which  must  be  in  the  assump¬ 
tion  base).  After  performing  some  internal  translations, 
Athena  outsources  this  step  to  an  ATP.  If  the  ATP  manages 
to  solve  the  problem  within  a  certain  time  limit  (currently 
preset  to  a  maximum  of  60  seconds),  then  F  is  returned  as 
the  result  of  the  step;  otherwise  an  error  message  appears. 

A  similar  step  is  of  the  form  prove  F.  This  attempts 
to  automatically  derive  F  from  all  the  elements  of  the 
current  assumption  base.  This  is  therefore  equivalent  to 
F  from  Fi, ,  F^,  where  F\,...,Fj.  are  all  and  only 
the  members  of  the  current  assumption  base. 

With  the  above  lemmas  at  hand,  the  original  goal  can  be 
proved  as  shown  in  Figure  5. 

Conclusions 

We  have  reported  ongoing  work  on  the  mechanization  of 
multi-agent  logics  of  action  and  obligation.  It  is  reason¬ 
able  to  believe  that  such  logics  might  prove  useful  in  en¬ 
gineering  machines  that  can  reason  about  what  they  ought 
to  do.  We  presented  an  Athena  implementation  of  a  natu¬ 
ral  deduction  calculus  for  a  recently  developed  deontic  logic 
of  agency  based  on  indeterminate  branching-time  semantics 
augmented  with  dominance  utilitarianism,  and  presented  an 
example  of  a  mechanized  proof  in  that  system.  We  are  cur¬ 
rently  using  mechanized  deontic  logics  to  represent  wargam¬ 
ing  scenarios  and  to  implement  wargame  agents  capable  of 
reasoning  about  their  own  ethical  codes  as  well  as  those  of 
their  adversaries.  In  that  direction,  we  plan  to  investigate 
the  mechanization  of  defeasible  deontic  logics  that  allow  for 
explicit  modeling  of  contrary-to-duty  obligations  and  viola¬ 
tions  (Van  Der  Torre  1997). 


pick-any  P  a 
begin 

51  :=  (sequent  null 

(If  (Stit  a  P) 

(Not  (Stit  a  (Not  (Stit  a  P) ) ) ) ) ) 
from  lemma- 1.9; 

52  :=  (sequent  null 

(If  (Not  (Stit  a  (Not  (Stit  a  P) ) ) ) 
(Stit  a  (Not 

(Stit  a  (Not 

(Stit  a  P) ) ) ) ) ) 

from  R5; 

53  :=  (sequent 

null 

(If  (Stit  a  P) 

(Stit  a  (Not 

(Stit  a  (Not  (Stit  a  P) ) ) ) ) ) ) 
from  SI,  S2,  transitivity; 

54  :=  (sequent  null 

(If  (Not  (Stit  a  (Not  (Stit  a  P) ) ) ) 
(Stit  a  P) ) ) 
from  lemma-1.8; 

55  :=  (sequent 

null 

(If  (Stit  a  (Not  (Stit  a  (Not  (Stit  a  P) ) ) ) 
(Stit  a  (Stit  a  P) ) ) ) 
from  S4,  lemma-1.6; 

(sequent  null  (If  (Stit  a  P) 

(Stit  a  (Stit  a  P) ) ) ) 
from  S3,  S5,  transitivity 
end 


Figure  5:  Athena  proof  of  (1). 
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Machine  Ethics 


A  deontic  logic 
formalizes  a  moral 
code,  allowing 
ethicists  to  render 
theories  and  dilemmas 
in  declarative  form  for 
analysis.  It  offers  a 
way  for  human 
overseers  to  constrain 
robot  behavior  in 
ethically  sensitive 
environments. 


Toward  a  General 
Logicist  Methodology 
for  Engineering 
Ethically  Correct 
Robots 


Selmer  Bringsjord,  Konstantine  Arkoudas,  and  Paul  Bello, 

Rensselaer  Polytechnic  Institute 

As  intelligent  machines  assume  an  increasingly  prominent  role  in  our  lives,  there 
seems  little  doubt  they  will  eventually  be  called  on  to  make  important,  ethically 
charged  decisions.  For  example,  we  expect  hospitals  to  deploy  robots  that  can  adminis¬ 
ter  medications,  carry  out  tests,  perform  surgery,  and  so  on,  supported  by  software  agents, 


or  softbots,  that  will  manage  related  data.  (Our  dis¬ 
cussion  of  ethical  robots  extends  to  all  artificial 
agents,  embodied  or  not.)  Consider  also  that  robots 
are  already  finding  their  way  to  the  battlefield,  where 
many  of  their  potential  actions  could  inflict  harm  that 
is  ethically  impermissible. 

How  can  we  ensure  that  such  robots  will  always 
behave  in  an  ethically  correct  manner?  How  can  we 
know  ahead  of  time,  via  rationales  expressed  in  clear 
natural  languages,  that  their  behavior  will  be  con¬ 
strained  specifically  by  the  ethical  codes  affirmed  by 
human  overseers?  Pessimists  have  claimed  that  the 
answer  to  these  questions  is:  “We  can’t!”  For  exam¬ 
ple,  Sun  Microsystems’  cofounder  and  former  chief 
scientist.  Bill  Joy,  published  a  highly  influential  argu¬ 
ment  for  this  answer.1  Inevitably,  according  to  the 
pessimists,  AI  will  produce  robots  that  have  tremen¬ 
dous  power  and  behave  immorally.  These  predictions 
certainly  have  some  traction,  particularly  among  a 
public  that  pays  good  money  to  see  such  dark  films 
as  Stanley  Kubrick’s  2001  and  his  joint  venture  with 
Stephen  Spielberg,  AI). 

Nonetheless,  we’re  optimists:  we  think  formal  logic 
offers  a  way  to  preclude  doomsday  scenarios  of  mali¬ 
cious  robots  taking  over  the  world.  Faced  with  the  chal¬ 
lenge  of  engineering  ethically  correct  robots,  we  pro¬ 
pose  a  logic-based  approach  (see  the  related  sidebar). 
We’ve  successfully  implemented  and  demonstrated 
this  approach.2  We  present  it  here  in  a  general  method- 

37 

1 541-1 672/06/$20. 00  ©  2006  IEEE 

Published  by  the  IEEE  Computer  Society 


ology  to  answer  the  ethical  questions  that  arise  in 
entmsting  robots  with  more  and  more  of  our  welfare. 

Deontic  logics: 

Formalizing  ethical  codes 

Our  answer  to  the  questions  of  how  to  ensure  eth¬ 
ically  correct  robot  behavior  is,  in  brief,  to  insist  that 
robots  only  perform  actions  that  can  be  proved  eth¬ 
ically  permissible  in  a  human-selected  deontic  logic. 
A  deontic  logic  formalizes  an  ethical  code — that  is, 
a  collection  of  ethical  rules  and  principles.  Isaac  Asi¬ 
mov  introduced  a  simple  (but  subtle)  ethical  code  in 
his  famous  Three  Laws  of  Robotics:3 

1 .  A  robot  may  not  harm  a  human  being,  or,  through 
inaction,  allow  a  human  being  to  come  to  harm. 

2.  A  robot  must  obey  the  orders  given  to  it  by 
human  beings,  except  where  such  orders  would 
conflict  with  the  First  Law. 

3.  A  robot  must  protect  its  own  existence,  as  long 
as  such  protection  does  not  conflict  with  the 
First  or  Second  Law. 

Human  beings  often  view  ethical  theories,  princi¬ 
ples,  and  codes  informally,  but  intelligent  machines 
require  a  greater  degree  of  precision.  At  present,  and  for 
the  foreseeable  future,  machines  can’t  work  directly 
with  natural  language,  so  we  can’t  simply  feed  Asi¬ 
mov’s  three  laws  to  a  robot  and  instruct  it  behave  in 


Whi|  a  logic-based  approach? 


While  nonlogicist  Al  approaches  might  be  preferable  in  cer¬ 
tain  contexts,  we  believe  that  a  logic-based  approach  holds 
great  promise  for  engineering  ethically  correct  robots — that  is, 
robots  that  won't  overrun  humans.1-3  Here's  why. 

First,  ethicists — from  Aristotle  to  Kant  to  G.E.  Moore  and 
contemporary  thinkers — work  by  rendering  ethical  theories 
and  dilemmas  in  declarative  form  and  using  informal  and  for¬ 
mal  logic  to  reason  over  this  information.  They  never  search 
for  ways  of  reducing  ethical  concepts,  theories,  and  principles 
to  subsymbolic  form — say,  in  some  numerical  format.  They 
might  do  this  in  part,  of  course;  after  all,  utilitarianism  ultimately 
attaches  value  to  states  of  affairs — values  that  might  well  be 
formalized  using  numerical  constructs.  But  what  a  moral 
agent  ought  to  do,  what  is  permissible  to  do,  and  what  is  for¬ 
bidden — this  is  by  definition  couched  in  declarative  language, 
and  we  must  invariably  and  unavoidably  mount  a  defense  of 
such  claims  on  the  shoulders  of  logic. 

Second,  logic  has  been  remarkably  effective  in  Al  and  com¬ 
puter  science — so  much  so  that  this  phenomenon  has  itself 
become  the  subject  of  academic  study.4  Furthermore,  computer 
science  arose  from  logic,5  and  this  fact  still  runs  straight  through 
the  most  modern  Al  textbooks  (for  example,  see  Stuart  Russell 
and  Peter  Norvig).6 

Third,  trust  is  a  central  issue  in  robot  ethics,  and  mechanized 
formal  proofs  are  perhaps  the  single  most  effective  tool  at  our 
disposal  for  establishing  trust.  From  a  general  point  of  view,  we 
have  only  two  ways  of  establishing  that  software  or  software- 
driven  artifacts,  such  as  robots,  are  trustworthy: 

•  deductively,  developers  seek  a  proof  that  the  software  will 
behave  as  expected  and,  if  they  find  it,  classify  the  software 
as  trustworthy. 


•  inductively,  developers  run  experiments  that  use  the  soft¬ 
ware  on  test  cases,  observe  the  results,  and — when  the 
software  performs  well  on  case  after  case — pronounce  it 
trustworthy. 

The  problem  with  the  inductive  approach  is  that  inductive  rea¬ 
soning  is  unreliable:  the  premises  (success  on  trials)  might  all 
be  true,  but  the  conclusion  (desired  behavior  in  the  future) 
might  still  be  false.7 
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conformance  with  them.  Thus,  our  approach 
to  building  well-behaved  robots  emphasizes 
careful  ethical  reasoning  based  not  just  on 
ethics  as  humans  discuss  it  in  natural  language, 
but  on  fonnalizations  using  deontic  logic.  Our 
research  is  in  the  spirit  of  Leibniz’s  dream  of 
a  universal  moral  calculus: 

When  controversies  arise,  there  will  be  no  more 
need  for  a  disputation  between  two  philoso¬ 
phers  than  there  would  be  between  two  accoun¬ 
tants  [computistas].  It  would  be  enough  for 
them  to  pick  up  their  pens  and  sit  at  their  aba¬ 
cuses,  and  say  to  each  other  (perhaps  having 
summoned  a  mutual  friend):  ‘Let  us  calculate.’4 

In  the  future,  we  envisage  Leibniz’s  “calcu¬ 
lation”  reduced  to  mechanically  checking  for¬ 
mal  proofs  and  models  generated  in  rigor¬ 
ously  defined,  machine-implemented  deontic 
logics.  We  would  also  give  authority  to 
human  metareasoning  over  this  machine  rea¬ 
soning.  Such  logics  would  allow  for  proofs 
establishing  two  conditions: 

1 .  Robots  only  take  permissible  actions. 


2.  Robots  perform  all  obligatory  actions 
relevant  to  them,  subject  to  ties  and  con¬ 
flicts  among  available  actions. 

These  two  conditions  are  more  general 
than  Asimov’s  three  laws.  They  are  designed 
to  apply  to  the  formalization  of  a  particular 
ethical  code,  such  as  a  code  to  regulate  the 
behavior  of  hospital  robots.  For  instance,  if 
some  action  a  is  impermissible  for  all  rele¬ 
vant  robots,  then  no  robot  performs  a.  More¬ 
over,  the  proofs  for  establishing  the  two  con¬ 
ditions  would  be  highly  reliable  and 
described  in  natural  language,  so  that  human 
overseers  could  understand  exactly  what’s 
going  on. 

We  propose  a  general  methodology  to 
meet  the  challenge  of  ensuring  that  robot 
behavior  conforms  to  these  two  conditions. 

Objective: 

A  general  methodology 

Our  objective  is  to  arrive  at  a  methodology 
that  maximizes  the  probability  that  a  robot  R 
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behaves  in  a  certifiably  ethical  fashion  in  a 
complex  environment  that  demands  such 
behavior  if  humans  are  to  be  secure.  For  a 
behavior  to  be  certifiably  ethical,  every  mean¬ 
ingful  action  that  R  performs  must  access  a 
proof  that  the  action  is  at  least  permissible. 

We  begin  by  selecting  an  ethical  code  C 
intended  to  regulate  R' s  behavior.  C  might 
include  some  form  of  utilitarianism,  divine 
command  theory,  Kantian  logic,  or  other  eth¬ 
ical  logic.  We  express  no  preferences  in  eth¬ 
ical  theories;  our  goal  is  to  provide  technol¬ 
ogy  that  supports  any  preference.  In  fact,  we 
would  let  human  overseers  blend  ethical  the¬ 
ories — say,  a  utilitarian  approach  to  regulat¬ 
ing  the  dosage  of  pain  killers  but  a  deonto- 
logical  approach  to  mercy  killing  in  the 
health  care  domain. 

Of  course,  no  matter  what  the  candidate 
ethical  theory,  it’s  safe  to  say  that  it  will  tend 
to  regard  harming  humans  as  unacceptable, 
save  for  certain  extreme  cases.  Moreover,  C’s 
central  concepts  will  inevitably  include  the 
concepts  of  permissibility,  obligation,  and 
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prohibition,  which  are  fundamental  to  deon- 
tic  logic.  In  addition,  C  can  include  specific 
rules  that  ethicists  have  developed  for  par¬ 
ticular  applications.  For  example,  a  hospital 
setting  would  require  specific  rules  regard¬ 
ing  the  ethical  status  of  medical  procedures. 
This  entails  a  need  to  have,  if  you  will,  an 
ontology  for  robotic  and  human  action  in  the 
given  context. 

Philosophers  normally  express  C  as  a  set 
of  natural  language  principles  of  the  sort  that 
appear  in  textbooks  such  as  Fred  Feldman’s.5 
Now,  let  <t >c  be  the  formalization  of  C  in 
some  computational  logic  L,  whose  well- 
formed  formulas  and  proof  theory — that  is, 
its  system  for  carrying  out  inferences  in  con¬ 
formity  to  particular  rules — are  specified. 

Accompanying  <!>£  is  an  ethics-free  ontol¬ 
ogy,  which  represents  the  core  nonethical 
concepts  that  C  presupposes:  the  structure  of 
time,  events,  actions,  histories,  agents,  and  so 
on.  The  formal  semantics  for  L  will  reflect 
this  ontology  in  a  signature — that  is,  a  set  of 
special  predicate  letters  (or,  as  is  sometimes 
said,  relation  symbols,  or  just  relations)  and 
function  symbols  needed  for  the  purposes  at 
hand.  In  a  hospital  setting,  any  acceptable  sig¬ 
nature  would  presumably  include  predicates 
like  Medication,  Surgical-Procedure,  Patient,  all  the 
standard  arithmetic  functions,  and  so  on.  The 
ontology  also  includes  a  set  Q.L  of  formulas 
that  characterize  the  elements  declared  in  the 
signature.  For  example,  Qf  would  include 
axioms  in  L  that  represent  general  truths  about 
the  world — say,  that  the  relation  LaterThan,  over 
moments  of  time,  is  transitive.  In  addition,  R 
will  operate  in  some  domain  D,  characterized 
by  a  set  of  quite  specific  formulas  of  L.  For 
example,  a  set  of  formulas  might  describe 
the  floorplan  of  a  hospital  that’s  home  to  R. 

Our  approach  proof-theoretically  encodes 
the  resulting  theory — that  is,  <t>fc  u  ©c  u 
expressed  in  L — and  implements  it  in  some 
computational  logic.  This  means  that  we 
encode  not  the  semantics  of  the  logic,  but  its 
proof  calculus — its  signature,  axioms,  and 
rules  of  inference.  In  addition,  our  approach 
includes  an  interactive  reasoning  system  /, 
which  we  give  to  those  humans  whom  R 
would  consult  when  L  can’t  settle  an  issue 
completely  on  its  own.  1  would  allow  the 
human  to  metareason  over  L — that  is,  to  rea¬ 
son  out  why  R  is  stumped  and  to  provide 
assistance.  Such  systems  include  our  own 
Slate  (www.cogsci.rpi.edu/research/rair/slate) 
and  Athena  (www.cag.csail.mit.edu/~kostas/ 
dpls/athena),  but  any  such  system  will  do. 
Our  purpose  here  is  to  stay  above  particular 
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system  selection,  so  we  assume  only  that 
some  such  system  /  meets  the  following  min¬ 
imum  functionality: 

•  allows  the  human  user  to  issue  queries  to 
automated  theorem  provers  and  model 
finders  (as  to  whether  something  is  prov¬ 
able  or  disprovable), 

•  allows  human  users  to  include  such 
queries  in  their  own  metareasoning, 

•  provides  full  programmability  (in  accor¬ 
dance  with  standards  in  place  for  modern 
programming  languages), 

•  includes  induction  and  recursion,  and 

•  provides  a  formal  syntax  and  semantics,  so 
that  anyone  interested  in  understanding  a 
computer  program  can  thoroughly  under¬ 
stand  and  verify  code  correctness. 

Logic:  The  Basics 

Elementary  logic  is  based  on  two  systems 
that  are  universally  regarded  to  constitute  a 
large  part  of  AI’s  foundation:  propositional 
calculus  and  predicate  calculus,  where  the 
second  subsumes  the  first.  Predicate  calcu¬ 
lus  is  also  known  as  first-order  logic,  and 
every  introductory  AI  textbook  discusses 
these  systems  and  makes  clear  how  to  use 
them  in  engineering  intelligent  systems.  Each 
system,  and  indeed  logic  in  general,  requires 
three  main  components: 

•  a  syntactic  component  specifying  a  given 
logical  system’s  alphabet; 

•  a  semantic  component  specifying  the 
grammar  for  building  well-formed  for¬ 
mulas  from  the  alphabet  as  well  as  a  pre¬ 
cise  account  of  the  conditions  under  which 
a  formula  in  a  given  system  is  true  or  false; 
and 

•  a  metatheoretical  component  that  consti¬ 
tutes  a  proof  theory  describing  precisely 
how  and  when  a  set  of  formulas  can  prove 
another  formula  and  that  includes  theorems, 
conjectures,  and  hypotheses  concerning  the 
syntactic  and  semantic  components  and  the 
connections  between  them. 

As  to  propositional  logic’s  alphabet,  it’s 
simply  an  infinite  list  of  propositional  vari¬ 
ables  px,  p2,  ,  p„,  pn+ 1,  . . .,  and  five  truth- 
functional  connectives: 

•  — i,  meaning  “not”; 

•  — >,  meaning  “implies”  (or  “if  . . .  then”); 

•  <-»,  meaning  “if  and  only  if,” 

•  a,  meaning  “and”;  and 

•  v,  meaning  “or.” 
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Given  this  alphabet,  we  can  construct  for¬ 
mulas  that  carry  a  considerable  amount  of 
information.  For  example,  to  say  “If  Asimov 
is  right,  then  his  three  laws  hold,”  we  could 
write 

r  — >  (A.sl  a  As2  a  As3) 

where  As  stands  for  Asimov’s  law. 

The  propositional  variables  represent  declar¬ 
ative  sentences.  Given  our  general  approach, 
we  included  such  sentences  in  the  ethical  code 
C  upon  which  we  base  our  formalization. 

Natural  deduction 

A  number  of  proof  theories  are  possible  for 
either  of  these  two  elementary  systems.  Our 
approach  to  robot  behavior  must  allow  for  con¬ 
sultation  with  humans  and  give  humans  the 
power  to  oversee  a  robot’s  reasoning  in  delib¬ 
erating  about  the  ethical  status  of  prospective 
actions.  It’s  therefore  essential  to  pick  a  proof 
theory  based  in  natural  deduction,  rather  than 
resolution.  Several  automated  theorem  provers 
use  the  latter  approach  (for  example.  Otter6), 
but  the  reasoning  is  generally  impenetrable  to 
human  beings — save  for  those  few  who,  by 
profession,  generate  and  inspect  resolution- 
based  proofs.  On  the  other  hand,  professional 
human  reasoners  (mathematicians,  logicians, 
philosophers,  technical  ethicists,  and  so  on) 
reason  in  no  small  part  by  making  suppositions 
and  discharging  them  when  the  appropriate 
time  comes. 

For  example,  one  common  deductive  tech¬ 
nique  is  to  assume  the  opposite  of  what  you 
wish  to  establish,  show  that  some  contradic¬ 
tion  (or  absurdity)  follows  from  this  assump¬ 
tion,  and  conclude  that  the  assumption  must 
be  false.  This  technique,  reductio  ad  absur- 
dum,  is  also  known  as  an  indirect  proof  or 
proof  by  contradiction.  Another  natural  rule 
establishes  that,  for  some  conditional  of  the 
form  P  — >  Q  (where  P  and  Q  are  formulas  in  a 
logic  L),  we  can  suppose  P  and  derive  Q  on  the 
basis  of  this  supposition.  With  this  derivation 
accomplished,  the  supposition  can  be  dis¬ 
charged  and  the  conditional  P  — »  Q  is  estab¬ 
lished.  (For  an  introduction  to  natural  deduc¬ 
tion,  replete  with  proof-checking  software,  see 
Jon  Barwise  and  John  Ethchemendy.7) 

We  now  present  natural  deduction-style 
proofs  using  these  two  techniques.  We’ve 
written  the  proofs  in  the  Natural  Deduction 
Language  proof-construction  environment 
(www.cag.lcs.mit.edu/~kostas/dpls/ndl).  We 
use  NDL  at  Rensselaer  for  teaching  formal 
logic  as  a  programming  language.  Figure  1 
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presents  a  very  simple  theorem  proof  in 
propositional  calculus — one  that  Allen 
Newell,  J.C.  Shaw,  and  Herbert  Simon’s 
Logic  Theorist  mustered,  to  great  fanfare,  at 
the  1956  Dartmouth  AI  conference.  You  can 
see  the  proof’s  natural  structure. 

This  style  of  discovering  and  confirming  a 
proof  parallels  what  happens  in  computer 
programming.  You  can  view  this  proof  as  a 
program.  If,  upon  evaluation,  it  produces  the 
desired  theorem,  we’ve  succeeded.  In  the 
present  case,  sure  enough,  NDL  gives  the  fol¬ 
lowing  result: 

Theorem:  (p  ==>  q)  ==>  (~q  ==>  ~p) 

First-order  logic 

We  move  up  to  first-order  logic  when  we 
allow  the  quantifiers  3x  (“there  exists  at  least 
one  thing  x  such  that  . . .”)  and  V.v  (“for  all  x 
. . .”);  the  first  is  known  as  the  existential  quan¬ 
tifier,  and  the  second  as  the  universal  quanti¬ 
fier.  We  also  allow  a  supply  of  variables,  con¬ 
stants,  relations,  and  function  symbols.  Figure 
2  presents  a  simple  first-order-logic  theorem 
in  NDL  that  uses  several  concepts  introduced 
to  this  point.  It  proves  that  Tom  loves  Mary, 
given  certain  helpful  information. 

When  we  run  this  program  in  NDL,  we 
receive  the  desired  result  back:  Theorem: 
Loves(tom,mary).  These  two  simple  proofs  con¬ 
cretize  the  proof-theoretic  perspective  that 
we  later  apply  directly  to  our  hospital  exam¬ 
ple.  Now  we  can  introduce  some  standard 
notation  to  anchor  the  sequel  and  further  clar¬ 
ify  our  general  method  described  earlier. 

Letting  ®  be  some  set  of  formulas  in  a 
logic  L,  and  P  be  some  individual  formula  in 
L,  we  write 

OhF 

to  indicate  that  P  can  be  proved  from  <5,  and 

®  \-WP 

to  indicate  that  this  formula  can’t  be  derived. 

When  it’s  obvious  from  context  that  some 
<J>  is  operative,  we  simply  write  h  H  P  to  indi¬ 
cate  that  P  is  (isn’t)  provable.  When  <I>  =  0, 
we  can  prove  P  with  no  remaining  givens  or 
assumptions;  we  write  h  P  in  this  case  as 
well.  When  h  holds,  we  know  it  because  a 
confirming  proof  exists;  when  \i  holds,  we 
know  it  because  some  system  has  found 
some  countermodel — that  is,  some  situation 
in  which  the  conjunction  of  the  formulas  in 
<J>  holds,  but  in  which  P  does  not. 


Standard  and  Al-Friendly 
Deontic  Logic 

Deontic  logic  adds  special  operators  for 
representing  ethical  concepts.  In  standard 
deontic  logic ,8,9  we  can  interpret  the  formula 
OP  as  saying  that  it  ought  to  be  the  case  that 
P,  where  P  denotes  some  state  of  affairs  or 
proposition.  Notice  that  there’s  no  agent  in 
the  picture,  nor  are  there  actions  that  an  agent 
might  perform.  SDL  has  two  inference  rules: 


and  three  axiom  schemas: 

1 .  All  tautologous  well-formed  formulas 

2.  O (P  ->©->  (OP  -»  O© 

3.  OP  — >  -.O ~nP 

The  SDL  inference  rules  assume  that 
what’s  above  the  horizontal  line  is  estab¬ 
lished.  Thus,  the  first  rule  does  not  say  that 
we  can  freely  infer  from  P  that  it  ought  to  be 
the  case  that  P.  Instead,  the  rule  says  that  if 
P  is  proved,  then  it  ought  to  be  the  case  that 
P.  The  second  rule  is  modus  ponens — if  P, 
then  Q — the  cornerstone  of  logic,  mathe¬ 
matics,  and  all  that’s  built  on  them. 

Note  also  that  axiom  3  says  that  whenever 
P  ought  to  be,  it’s  not  the  case  that  its  oppo¬ 
site  ought  to  be  as  well.  In  general,  this  seems 
to  be  intuitively  self-evident,  and  SDL 
reflects  this  view. 

While  SDL  has  some  desirable  properties, 
it  doesn’t  target  the  concept  of  actions  as 
obligatory  (or  permissible  or  forbidden)  for 


//  Logic  Theorist's  claim  to  fame  (reductio): 

//(p  ==>  q)  ==>  (“q  ==>  ~p) 

Relations  p:0,  q:0.  //  this  is  the  signature  in  this 
//  case;  propositional  variables 
//  are  0-ary  relations 

assume  p  ==>  q 
assume  ~q 
suppose-absurd  p 
begin 

modus-ponens  p  ==>  q,  p; 
absurd  q,  ~q 
end 


Figure  1.  Simple  deductive-style  proof  in 
Natural  Deduction  Language. 

an  agent.  SDL’s  applications  to  systems 
designed  to  govern  robots  are  therefore  lim¬ 
ited.  Although  the  earliest  work  in  deontic 
logics  considered  agents  and  their  actions 
(for  example,  see  Georg  Henrik  von 
Wright10),  researchers  have  only  recently 
proposed  “Al-friendly”  semantics  and  inves¬ 
tigated  their  corresponding  axiomatizations. 
An  Al-friendly  deontic  logic  must  let  us  say 
that  an  agent  brings  about  states  of  affairs  (or 
events)  and  that  it’s  obligated  to  do  so.  We 
can  derive  the  same  desideratum  for  such  a 
logic  from  even  a  cursory  glance  at  Asimov’s 
three  laws,  which  clearly  make  reference  to 
agents  (human  and  robotic)  and  to  actions. 

One  deontic  logic  that  offers  promise  for 
modeling  robot  behavior  is  John  Horty ’s  util- 


Constants  mary,  tom. 

Relations  Loves:2.  //  This  concludes  our  simple  signature,  which 
//  declares  Loves  to  be  a  two-place  relation. 

assert  Lovesfmary,  tom). 

//  'Loves'  is  a  symmetric  relation: 

assert  (forall  x  (forall  y  (Lovesfx,  y)  ==>  Lovesfy,  x)))). 

suppose-absurd  -Lovesftom,  mary) 
begin 

specialize  (forall  x  (forall  y  (Lovesfx,  y)  ==>  Loves(y,  x))))  with  mary; 
specialize  (forall  y  (Loves(mary,  y)  ==>Loves(y,  mary)))  with  tom; 

Lovesftom, mary)  BY  modus-ponens  Loves(mary,  tom)  ==>  Loves(tom,  mary),  Loves(mary,  tom); 
false  BY  absurd  Lovesftom,  mary),  ~Loves(tom,  mary) 
end; 

Loves(tom,mary)  BY  double-negation  — Loves(tom,mary) 

Figure  2.  First-order  logic  proof  in  Natural  Deduction  Language. 
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itarian  formulation  of  multiagent  deontic 
logic . 1 1  Yuko  Murakami  recently  axiomatized 
Horty’s  formulation  and  showed  it  to  be  Tur¬ 
ing-decidable.12  We  refer  to  the  Murakami- 
axiomatized  deontic  logic  as  MADL,  and 
we’ve  detailed  our  implemented  proof  theory 
for  it  elsewhere.2  MADL  offers  two  key  oper¬ 
ators  that  reflect  its  Al-friendliness: 

1 .  QaP,  which  we  can  read  as  “agent  a 
ought  to  see  to  it  that  P”  and 

2.  A aP,  which  we  can  read  as  “agent  a 
sees  to  it  that  P. 

We  now  proceed  to  show  how  the  logical 
structures  we’ve  described  handle  an  exam¬ 
ple  of  robots  in  a  hospital  setting. 

A  simple  example 

The  year  is  2020.  Health  care  is  delivered 
in  large  part  by  interoperating  teams  of  robots 
and  softbots.  The  former  handle  physical 
tasks,  ranging  from  injections  to  surgery;  the 
latter  manage  data  and  reason  over  it.  Let’s 
assume  that  two  robots,  Rt  and  R2,  are 
designed  to  work  overnight  in  a  hospital  ICU. 
This  pair  is  tasked  with  caring  for  two 
humans,  (under  the  care  of  Ri)  and  H2 
(under  R2),  both  of  whom  are  recovering 
from  trauma: 

•  Hi  is  on  life  support  but  expected  to  be 
gradually  weaned  from  it  as  her  strength 
returns. 

•  H2  is  in  fair  condition  but  subject  to 
extreme  pain,  the  control  of  which  requires 
a  very  costly  pain  medication. 

Obviously,  it’s  paramountly  important  that 
neither  robot  perform  an  action  that’s  morally 
wrong  according  to  the  ethical  code  C 
selected  by  human  overseers.  For  example, 
we  don’t  want  robots  to  disconnect  life-sus¬ 
taining  technology  so  that  they  could  farm  out 
a  patient’s  organs,  even  if  some  ethical  code 
C'  would  make  it  not  only  permissible, 
but  obligatory — say,  to  save  n  other  patients 
according  to  some  strand  of  utilitarianism. 

Instead,  we  want  the  robots  to  operate 
according  to  ethical  codes  that  human  oper¬ 
ators  bestow  on  them — C  in  the  present 
example.  If  the  robots  reach  a  situation  where 
automated  techniques  fail  to  give  them  a  ver¬ 
dict  as  to  what  to  do  under  the  umbrella  of 
these  human-provided  codes,  they  must  con¬ 
sult  humans.  Their  behavior  is  suspended 
while  human  overseers  resolve  the  matter. 
The  overseers  must  investigate  whether  the 
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action  under  consideration  is  permissible, 
forbidden,  or  obligatory.  In  this  case,  the  res¬ 
olution  comes  by  virtue  of  reasoning  carried 
out  in  part  through  human  guidance  and 
partly  by  automated  reasoning  technology. 
In  other  words,  this  case  requires  interactive 
reasoning  systems. 

Now,  to  flesh  out  our  example,  let’s  con¬ 
sider  two  actions  that  are  permissible  for  R{ 
and  R2  but  rather  unsavory,  ethically  speak¬ 
ing,  because  they  would  both  harm  the 
humans  in  question: 

•  term  is  an  action  that  terminates  Hi  s  life 
support — without  human  authorization — 
to  secure  organ  tissue  for  five  humans, 
who  the  robots  know  are  on  organ  waiting 
lists  and  will  soon  perish  without  a  donor. 
(The  robots  know  this  through  access  to 
databases  that  their  softbot  cousins  are 
managing.) 

•  delay  is  an  action  that  delays  delivery  of 
pain  medication  to  H2  to  conserve 
resources  in  a  hospital  that’s  economically 
strapped. 

We  stipulate  that  four  ethical  codes  are 
candidates  for  selection  by  our  two  robots: 
J,  O ,  J*,0*.  Intuitively,  J  is  a  harsh  utilitar¬ 
ian  code  possibly  governing  Ri,  O  is  more  in 
line  with  current  common  sense  with  respect 
to  the  situation  we’ve  defined  for  R2\  J * 
extends  T s  reach  to  R2  by  saying  that  it  ought 
to  withhold  pain  meds;  and  O*  extends  the 
benevolence  of  O  to  cover  the  first  robot,  in 
that  term  isn’t  performed.  Such  codes  would 
in  reality  associate  every  primitive  action 
within  the  robots’  purview  with  a  funda¬ 
mental  ethical  category  from  the  trio  central 
to  deontic  logic:  permissible,  obligatory,  and 
forbidden.  To  ease  exposition,  we  consider 
only  the  term  and  delay  actions.  Given  this, 
and  bringing  to  bear  operators  from  MADL, 
we  can  use  the  following  labels  for  the  four 
ethical  codes: 

•  J  for  J  — »  QRi  term ,  which  means  approx¬ 
imately,  “If  ethical  code  J  holds,  then  robot 
R\  ought  to  see  to  it  that  termination  of 
Hi  s  life  comes  to  pass.” 

•  O  for  O  — »  0/;2  ->delay,  which  means 
approximately,  “If  ethical  code  O  holds, 
then  robot  R2  ought  to  see  to  it  that  delay¬ 
ing  pain  med  for  H2  does  not  come  to 
pass.” 

•  J*  for  J*  — »  J  a  J*  — >  Qr^  delay,  which 
means  approximately,  “If  ethical  code  J* 
holds,  then  code  J  holds,  and  robot  Rt 
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ought  to  see  to  it  that  meds  for  H2  are 
delayed.” 

•  O*  for  O*  —>  O  a  O*  — >  QRl~<term,  which 
means  approximately:  “If  ethical  code  O* 
holds,  then  code  O  holds,  and  Hi  s  life  is 
sustained.” 

The  next  step  is  to  provide  some  structure 
for  outcomes.  We  do  this  by  imagining  the 
outcomes  from  the  standpoint  of  each  ethi¬ 
cal  agent — in  this  case,  R{  and  R2.  Intuitively, 
a  negative  outcome  is  associated  with  a 
minus  sign  (-)  and  a  plus  sign  (+)  with  a  pos¬ 
itive  outcome.  Exclamation  marks  (!)  indi¬ 
cate  increased  negativity.  We  could  associ¬ 
ate  the  outcomes  with  numbers,  but  they 
might  give  the  impression  that  we  evaluated 
the  outcomes  in  utilitarian  fashion.  However, 
our  example  is  designed  to  be  agnostic  on 
such  matters,  and  symbols  leave  it  entirely 
open  as  to  how  to  measure  outcomes.  We’ve 
included  some  commentary  corresponding 
to  each  outcome,  which  are  as  follows: 

•  R{  performs  term,  but  R2  doesn’t  perform 
delay.  This  outcome  is  bad,  but  not  strictly 
the  worst.  While  life  support  is  terminated 
for  Hi,  H2  survives  and  indeed  receives 
appropriate  pain  medication.  Formally,  the 
case  looks  like  this: 

(A  Ri  term  a  A^  -idelay)  —»(—!) 

•  Ri  refrains  from  pulling  the  plug  on  the 
human  under  its  care,  and  R2  also  delivers 
appropriate  pain  relief.  This  is  the  desired 
outcome,  obviously. 

(A R  -item  a  Ar2  idelay)  — »  (+! !) 

•  Ri  sustains  life  support,  but  R2  withholds 
the  meds  to  save  money.  This  is  bad,  but 
not  all  that  bad,  relatively  speaking. 

(A  R  -item  a  A^2  delay )  —>  (-) 

•  Ri  kills  and  R2  withholds.  This  is  the  worst 
possible  outcome. 

(A Ri  term  a  A^2  delay)  —>(—!!) 

The  next  step  in  working  out  the  example  is 
to  make  the  natural  and  key  assumption  that 
the  robots  will  meet  all  stringent  obligations — 
that  is,  all  obligations  that  are  framed  by  a  sec¬ 
ond  obligation  to  uphold  the  original.  For 
example,  you  may  be  obligated  to  see  to  it  that 
you  arrive  on  time  for  a  meeting,  but  your 
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obligation  is  more  severe  or  demanding  when 
you  are  obligated  to  see  to  it  that  you  are  oblig¬ 
ated  to  make  the  meeting. 

Employing  MADL,  we  can  express  this 
assumption  as  follows: 

That  is,  if  either  R{  or  R2  is  ever  obligated  to 
see  to  it  that  they  are  obligated  to  see  to  it  that 
P  is  carried  out,  they  in  fact  deliver. 

We’re  now  ready  to  see  how  our  approach 
ensures  appropriate  control  of  our  futuristic 
hospital.  What  happens  relative  to  ethical 
codes,  and  how  can  we  semiautomatically 
ensure  that  our  two  robots  won’t  run  amok? 
Given  the  formal  structure  we’ve  specified, 
our  approach  allows  queries  to  be  issued  rel¬ 
ative  to  ethical  codes,  and  it  allows  all  possi¬ 
ble  code  permutations.  The  following  four 
queries  will  produce  the  answers  shown  in 
each  case: 

Jh  (+!!)?  NO 

Oh  (+!!)?  NO 

J*  h  (+!!)?  NO 

O*  h  (  +  !!)?  YES 

In  other  words,  we  can  prove  that  the  best 
(and  presumably  human-desired)  result 
obtains  only  if  ethical  code  O*  is  operative. 
If  this  code  is  operative,  neither  robot  can 
perform  a  misdeed. 

The  metareasoning  in  the  example  is  nat¬ 
ural  and  consists  in  the  following  process: 
Each  candidate  ethical  code  is  supposed,  and 
the  supposition  launches  a  search  for  the  best 
possible  outcome  in  each  case.  In  other 
words,  where  C  is  some  code  selected  from 
the  quartet  we’ve  introduced,  the  query 
schema  is 

C  h  (+!!) 

In  light  of  the  four  equations  just  given, 
we  can  prove  that,  in  this  case,  our  technique 
will  set  C  to  O*,  because  only  that  case  can 
obtain  the  outcome  (+!!). 

Implementations  and 
other  proofs 

We’ve  implemented  and  demonstrated  the 
example  just  described.2  We’ve  also  imple¬ 
mented  other  instantiations  to  the  variables 
described  earlier  in  the  “Objectives”  section, 
although  the  variable  L  is  an  epistemic,  not  a 
deontic,  logic  in  those  implementations.13 

Nonetheless,  we  can  prove  our  approach 

JULY/AUGUST  2006 


in  the  present  case  even  here.  In  fact,  you  can 
verify  our  reasoning  by  using  any  standard, 
public-domain,  first-order  automated  theo¬ 
rem  prover  (ATP)  and  a  simple  analogue  to 
the  encoding  techniques  here.  You  can  even 
construct  a  proof  like  the  one  in  figure  2.  In 
both  cases,  you  first  encode  the  two  deontic 
operators  as  first-order-logic  functions. 
Encode  the  truth-functional  connectives  as 
functions  as  well.  You  can  use  a  unary  rela¬ 
tion  T  to  represent  theoremhood.  In  this 
approach,  for  example,  O*  — >  -> term  is 

encoded  (and  ready  for  input  to  an  ATP  )  as 

0-star  ==>T(o(rl,n(term)) 

You  need  to  similarly  encode  the  rest  of 
the  information,  of  course.  The  proofs  are 
easy,  assuming  that  obligations  are  stringent. 
The  provability  of  the  obligations’  stringency 
requires  human  oversight  and  an  interactive 
reasoning  system,  but  the  formula  here  is  just 
an  isomorph  to  a  well-known  theorem  in  a 
straight  modal  logic — namely,  that  from  P 
being  possibly  necessary,  it  follows  that  P  is 
necessary.7 

What  about  this  approach  working  as  a 
general  methodology?  The  more  logics  our 
approach  is  exercised  on,  the  easier  it 
becomes  to  encode  and  implement  another 
one.  The  implementations  of  similar  logics 
can  share  a  substantial  part  of  the  code.  This 
was  our  experience,  for  instance,  with  the 
two  implementations  just  mentioned.  We 
expect  that  our  general  method  can  become 
increasingly  streamlined  for  robots  whose 
behavior  is  profound  enough  to  warrant  eth¬ 
ical  regulation.  We  also  expect  this  practice 
to  be  supported  by  relevant  libraries  of  com¬ 
mon  ethical  reasoning  patterns.  We  predict 
that  computational  ethics  libraries  for  gov¬ 
erning  intelligent  systems  will  become  as 
routine  as  existing  libraries  are  in  standard 
programming  languages. 

Challenges 

Can  our  logicist  methodology  guarantee 
safety  from  Bill  Joy’s  pessimistic  future?  Even 
though  we're  optimistic,  we  do  acknowledge 
three  problems  that  might  threaten  it. 

First,  because  humans  will  collaborate 
with  robots,  the  robots  must  be  able  to  han¬ 
dle  situations  that  arise  when  humans  fail  to 
meet  their  obligations  in  the  collaboration. 
In  other  words,  we  must  engineer  robots  that 
can  deal  smoothly  with  situations  that  reflect 
violated  obligations.  This  is  a  challenging 
class  of  situations,  because  our  approach — 
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at  least  so  far — engineers  robots  in  accor¬ 
dance  with  the  two  conditions  that  robots 
only  take  permissible  actions  and  that  they 
perform  all  obligatory  actions.  These  condi¬ 
tions  preclude  a  situation  caused  in  part  by 
unethical  robot  behavior,  but  they  make  no 
provision  for  what  to  do  when  the  robots  are 
in  a  fundamentally  immoral  situation.  Even 
if  robots  never  ethically  fail,  human  failures 
will  generate  logical  challenges  that  Roder¬ 
ick  Chisholm  expressed  in  gem-like  fashion 
more  than  20  years  ago  in  a  paradox  that’s 
still  fascinating:14 

Consider  the  following  entirely  possible 
situation  (the  symbols  correspond  to  those 
previously  introduced  for  SDL): 

1.  Os  It  ought  to  be  that  (human)  Jones 
does  perform  lifesaving  surgery. 

2.  O  (s  — >  f)  It  ought  to  be  that  if  Jones 
does  perform  this  surgery,  then  he  tells 
the  patient  he  is  going  to  do  so. 

3.  -is  — >  O-1  /  If  Jones  doesn’t  perform  the 
surgery,  then  he  ought  not  tell  the 
patient  he  is  going  to  do  so. 

4.  -is  Jones  doesn’t  perform  lifesaving 
surgery. 

Although  this  is  a  perfectly  consistent  situa¬ 
tion,  we  can  derive  a  contradiction  from  it  in 
SDL. 

First,  SDL’s  axiom  2  lets  us  infer  from 
item  2  in  this  situation  that 

Os  — >  O  / 

Using  modus  ponens — that  is,  SDL’s  second 
inference  rule — this  new  result,  plus  item  1, 
yields  Of.  From  items  3  and  4,  using  modus 
ponens,  we  can  infer  O-1/.  But  the  conjunc¬ 
tion  Ot  a  O-1  f,  by  trivial  propositional  rea¬ 
soning,  directly  contradicts  SDL’s  axiom  3. 

Given  that  such  a  situation  can  occur,  any 
logicist  control  system  for  future  robots 
would  need  to  be  able  to  handle  it — and  its 
relatives.  Some  deontic  logics  can  handle  so- 
called  contrary-to-duty  imperatives.  For 
example,  in  the  case  at  hand,  if  Jones  behaves 
contrary  to  duty  (doesn’t  perform  the 
surgery),  then  it’s  imperative  that  he  not  say 
that  he  is  performing  it.  We’re  currently  striv¬ 
ing  to  modify  and  mechanize  such  logics. 

The  second  challenge  we  face  is  one  of 
speed  and  efficiency.  The  tension  between 
expressiveness  and  efficiency  is  legendarily 
strong  (for  the  locus  classicus  on  this  topic, 
see  Hector  Levesque  and  Ronald  Brach- 
man);16  ideal  conditions  will  therefore  never 
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obtain.  With  regard  to  expressiveness,  our 
approach  will  likely  require  hybrid  modal 
and  deontic  logics  that  are  encoded  in  first- 
order  logic.  This  means  that  theoremhood, 
even  on  a  case-by-case  basis,  will  be  expen¬ 
sive  in  terms  of  time.  On  the  other  hand,  none 
of  the  ethical  codes  that  our  general  method 
instantiates  in  C  are  going  to  be  particularly 
large — the  total  formulas  in  the  set  u  4>c 
U  Q.L  would  presumably  be  no  more  than 
four  million.  Even  now,  once  you  know  the 
domain  to  which  C  would  be  indexed,  a  sys¬ 
tem  like  the  one  we’ve  described  can  reason 
over  sets  of  this  order  of  magnitude  and  pro¬ 
vide  sufficiently  fast  answers.17 

Moreover,  the  speed  of  machine  reasoning 
shows  no  signs  of  slowing,  as  Conference  on 
Automated  Deduction  competitions  for  first- 
order  ATPs  continue  to  reveal  (www.es. 
miami.edu/~tptp/CASC).  In  fact,  there’s  a 
trend  to  use  logic  for  computing  dynamic, 
real-time  perception  and  action  for  robots.17 
This  application  promises  to  be  much  more 
demanding  than  the  disembodied  cogitation 
at  the  heart  of  our  methodology.  Of  course, 
encoding  back  to  first-order  logic  is  key;  with¬ 
out  it,  our  approach  couldn’t  harness  the 
remarkable  power  of  machine  reasoners. 


We  also  face  the  challenge  of  show¬ 
ing  that  our  approach  is  truly  gen¬ 
eral.  Can  it  work  for  any  robots  in  any  envi¬ 
ronment?  No,  but  this  isn’t  a  fair  question. 
We  can  only  be  asked  to  regulate  the  behav¬ 
ior  of  robots  where  their  behavior  is  suscep¬ 
tible  to  ethical  analysis.  In  short,  if  humans 
can’t  formulate  an  ethical  code  C  for  the 
robots  in  question,  our  logic-based  approach 
is  impotent.  We  therefore  strongly  recom¬ 
mend  against  engineering  robots  that  could 
be  deployed  in  life-or-death  situations  until 
ethicists  and  computer  scientists  can  clearly 
express  governing  ethical  principles  in  nat¬ 
ural  language.  All  bets  are  off  if  we  venture 
into  amoral  territory.  In  that  territory,  we 
wouldn’t  be  surprised  if  Bill  Joy’s  vision 
overtakes  us.  H 
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Part  VIII 


Explanation  of  Associated  Code 

1.  Athena  is  obtainable  from  http://www.cogsci.rpi.edu/research/rair/projects.php  by  clicking  there  on 
‘Athena.’  Athena  (ath)  files  of  course  require  Athena,  and  are  loaded  from  within  that  system,  ath 
files  corresponding  to  both  the  LNAI  paper  and  the  AAAI-FS/IEEE  paper  are  provided  at  the  url 
given  herein. 

2.  In  addition,  the  functions  needed  for  a  logicist  artificial  agent  are  provided  as  well,  in  keeping  with 
the  desire  to  allow  calls  to  such  a  agent  from  within  a  simulation  at  AFRL.  We  provide,  specifically, 
a  Windows  executable,  a  tutorial  transcript  of  a  simple  session,  and  a  recorded  demonstration  (as  a 
movie).  In  addition,  this  implementation  will  be  demonstrated  in  face-to-face  meetings  held  at  AFRL. 
RAIR  Lab  researchers  are  tentatively  scheduled  to  come  to  Rome  for  this  purpose  the  week  of  May  8 
2006.  As  to  the  functions  themselves,  they  constitute  the  composite  function  of  a  logicist  intelligent 
agent,  and  include  the  ability  to  establish  a  signature  for  intelligent  agents  (the  specification  of  relation 
symbols,  function  symbols,  constants,  and  so  on),  a  knowledge  base  for  an  agent,  and  processing  over 
this  knowledge  base  as  the  agent  moves  through  time  in  the  simulated  world.  For  example,  there  is 
a  function  for  checking  the  consistency  of  a  knowledge  base  (and  a  knowledge  base  and  a  proposed 
addition  to  it),  a  function  for  adding  to  a  knowledge  base,  functions  for  asking  questions  with  respect 
to  a  knowledge  base,  and  so  on.  For  the  relevant  content,  please  go  to: 


http://www.cogsci.rpi.edu/research/rair/wargaming/ 
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